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ABSTRACT 


This  document  contains  the  three  volumes  of  the  technical 
report  for  development  of  the  prototype  of  the  Occupational  Information 
Network  (0*NET) , which  is  intended  to  replace  the  "Dictionary  of  Occupational 
Titles."  "General  Introduction"  (Norman  G.  Peterson)  presents  an  overview  of 
0*NET 1 s purpose,  content,  and  structure.  "Research  Method:  Development  and 
Field  Testing  of  the  Content  Model"  (Norman  G.  Peterson,  Michael  D.  Mumford, 
Kerry  Y.  Levin,  Jim  Green,  Joseph  Waksberg)  details  the  methodology  used  to 
assess  the  meaningfulness  and  usefulness  of  the  0*NET  system.  The  remaining 
four  papers  in  volume  1 describe  the  results  of  the  analysis  of  the  data  for 
the  following  0*NET  domains:  skills,  knowledges,  occupational  preparation, 
and  generalized  work  activities.  The  papers  are  as  follows:  "Basic  and 
Cross-Functional  Skills:  Evidence  for  the  Reliability  and  Validity  of  the 
Measures"  (Michael  D.  Mumford,  Norman  G.  Peterson,  Ruth  A.  Childs); 
"Knowledges:  Evidence  for  the  Reliability  and  Validity  of  the  Measures" 

(David  P.  Costanza,  Edwin  A.  Fleishman,  Joanne  C.  Marshall-Miles); 

"Education,  Training,  Experience,  and  Licensure/Certification:  Evidence  for 
the  Reliability  and  Validity  of  Measures"  (Lance  E.  Anderson);  and 
"Generalized  Work  Activities:  Evidence  for  the  Reliability  and  Validity  of 
Measures"  (Walter  C.  Borman,  P.  Richard  Jeanneret,  U.  Christean  Kubisiak, 


O 


Mary  Ann  Hanson).  Five  papers  contain  substantial  bibliographies.  The  five 
papers  in  volume  2 describe  the  results  of  the  analysis  of  the  data  for  the 
following  0*NET  domains:  work  context,  organizational  context,  abilities, 
occupational  values,  and  work  styles.  The  papers  are  as  follows:  "Work 

Context:  Evidence  for  the  Reliability  and  Validity  of  the  Measures"  (Mark  H. 
Strong,  P.  Richard  Jeanneret,  S.  Morton  McPhail,  Barry  R.  Blakely); 
"Organizational  Context:  Evidence  for  the  Reliability  and  Validity  of  the 
Measures"  (Sharon  Arad,  Mary  Ann  Hanson,  Robert  J.  Schneider);  "Abilities: 
Evidence  for  the  Reliability  and  Validity  of  the  Measures"  (Edwin  A. 
Fleishman,  David  P.  Costanza,  Joanne  C.  Marshall-Miles);  "Occupational 
Interests  and  Values:  Evidence  for  the  Reliability  and  Validity  of  the 
Occupational  Interest  Codes  and  the  Values  Measures"  (Christopher  E.  Sager), 
and  "Work  Style  Descriptors:  Evidence  for  the  Reliability  and  Validity  of 
Measures"  (Walter  C.  Borman,  U.  Christean  Kubisiak) . Four  papers  contain 
substantial  bibiographies.  Papers  in  volume  3 cover  topics  that  cut  across 
the  0*NET  domains,  including  possible  covariates  of  0*NET  rating  data, 
cross-domain  analyses,  cluster  analyses  of  occupations,  use  of  0*NET  data  to 
describe  occupations,  and  analyses  and  methods  of  collecting 
occupation-specific  information.  The  papers  are  as  follows:  "Occupational 
Descriptor  Covariates:  Possible  Sources  of  Variance  in  0*NET  Ratings"  (Ruth 
A.  Childs,  Norman  G.  Peterson,  Michael  D.  Mumford) ; "Cross-Domain  Analysis 
Results"  (Mary  Ann  Hanson,  Walter  C.  Borman,  U.  Christean  Kubisiak, 
Christopher  E.  Sager);  "Occupational  Classification:  Using  Basic  and 
Cross-Functional  Skills  and  Generalized  Work  Activities  to  Create  Job 
Families"  (Dwayne  G.  Norris,  Wayne  A.  Baughman,  Ashley  E.  Cooke,  Norman  G. 
Peterson,  Michael  D.  Mumford);  "Issues  in  0*NET  Applications"  (Walter  C. 
Borman,  Mary  Ann  Hanson,  U.  Christean  Kubisiak) ; and  "Occupation-Specific 
Descriptors:  Approaches,  Procedures,  and  Findings"  (Michael  D.  Mumford, 
Christopher  E.  Sager,  Wayne  A.  Baughman,  Ruth  A.  Childs);  and  "Conclusions 
and  Recommendations"  (Norman  G.  Peterson) . The  computer-assisted  telephone 
interview  protocol  for  organizational  representatives  and  other  data 
collection  materials  are  appended.  One  paper  contains  a substantial 
bibliography.  (MN) 
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The  American  Institutes  for  Research  and  its  subcontractors.  Personnel  Decisions 
Research  Institutes,  Inc.,  Management  Research  Institute,  Inc.,  Jeanneret  & Associates,  Inc.,  and 
Westat,  Inc.,  performed  the  work  described  herein  under  Contract  Number  94-542,  administered 
by  the  Utah  Department  of  Employment  Security,  on  behalf  of  the  U.S.  Department  of  Labor. 

Under  this  contract,  the  American  Institutes  for  Research  and  its  subcontractors 
developed  an  operational  prototype  for  an  occupational  data  collection,  analysis,  and 
dissemination  system-the  Occupational  Information  Network  or  0*NET~to  replace  the 
Department  of  Labor’s  Dictionary  of  Occupational  Titles.  This  report,  0*NET  Final  Technical 
Report,  submitted  by  the  American  Institutes  for  Research  as  a major  deliverable  under  this 
contract,  describes  the  empirical  evidence  provided  by  the  preliminary  data  collection  effort  for 
the  meaningfulness  of  the  prototype  system.  An  earlier  report.  Development  of  Prototype 
Occupational  Information  Network  (0*NET)  Content  Model,  described  the  development  of  the 
model  underlying  the  0*NET  and  the  design  of  the  questionnaires  used  to  collect  the 
occupational  information.  A separate  report,  0*NET:  An  Information  System  for  the  Workplace. 
Designing  an  Electronic  Infrastructure  (Rose,  Hesse,  Silver,  & Dumas,  1996),  describes  the 
development  of  the  electronic  database  and  provides  technical  documentation  for  the  database. 

The  Holland  Occupational  Codes  and  explanatory  text  included  in  Chapter  10  of  this 
document  are  adapted  and  reproduced  by  special  permission  of  the  Publisher,  Psychological 
Assessment  Resources,  Inc.,  Odessa,  FL  33556,  from  the  Dictionary  of  Holland  Occupational 
Codes,  Second  Edition  by  Gary  D.  Gottffedson,  Ph.D.,  and  John  L.  Holland,  Ph.D.,  Copyright, 
1982, 1989.  Further  reproduction  for  any  purpose  or  by  any  means  is  prohibited  without  the  prior 
written  permission  of  the  Publisher. 

Please  note  that  the  analysis  results  tables  in  Chapters  3 through  11  are  numbered 
uniformly  across  chapters.  Because  some  analyses  are  not  appropriate  for  every  domain, 
some  domains  are  missing  certain  table  numbers.  This  is  intentional.  Please  refer  to 
chapter  2 and,  particularly,  Figure  2-17  for  a listing  of  what  tables  should  appear  in  each 
chapter. 
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Chapter  1 

General  Introduction 

Norman  G.  Peterson 
American  Institutes  for  Research 


This  is  the  final  technical  report  for  the  development  of  the  prototype  of  the  Occupational 
Information  Network,  or  0*NET.  0*NET  is  intended  to  be  the  replacement  of  the  Dictionary  of 
Occupational  Titles  (DOT;  U.S.  Department  of  Labor,  1991).  This  project  has  its  origins  in  the 
report  of  the  Secretary  of  Labor’s  Advisory  Panel  on  the  DOT  (APDOT;  U.S.  Department  of 
Labor,  1993).  The  findings  of  that  committee  led  directly  to  this  effort.  Among  the 
recommendations  of  APDOT,  perhaps  most  seminal  were  the  general  recommendations  that  the 
DOT  replacement  be  an  electronic  database,  rather  than  a book,  and  that  it  contain  a 
comprehensive  set  of  occupational  information  that  would  meet  the  requirements  of  a diverse  set 
of  informational  consumers.  More  specifically,  the  APDOT  outlined  some  of  the  types  of 
occupational  descriptors  that  might  be  used  in  the  DOT  replacement,  recommended  the  use  of 
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survey  methodology,  and  called  for  frequent  updating  of  the  information  in  the  occupational 
database. 

Much  of  the  initial  effort  of  this  project  was  directed  toward  devising  a general  approach 
for  organizing  occupational  description  to  accomplish  these  ends.  The  results  of  those  efforts  are 
contained  in  the  “content  model”  report  (Peterson,  Mumford,  Boiman,  Jeanneret,  & Fleishman, 
1995).  Figure  1-1  shows  the  0*NET  content  model  as  described  in  that  report.  We  will  not 
attempt  to  duplicate  that  report  here,  but  several  points  about  the  model  bear  mention. 

First,  there  are  several  taxonomic  domains.  These  can  be  thought  of  as  multiple 
“windows”  for  viewing  an  occupation.  Thus,  an  experienced  worker  with  a long  work  history 
might  be  more  interested  in  cross-functional  skills  required  for  an  occupation,  while  a recent 
high-school  graduate  might  be  more  interested  in  abilities  or  basic  skills-since  the  recent 
graduate  will  have  had  less  time  and  fewer  experiences  to  develop  such  skills.  Vocational 
counselors  will  be  more  interested  in  the  interests’  and  values’  profiles  for  an  occupation,  while 
educators  might  be  more  interested  in  the  required  knowledges. 

The  use  of  multiple  windows  for  diverse  users  requires  the  development  of  relatively 
complete  taxonomies  within  each  window.  Thus,  users  do  not  have  to  access  many  windows  or 
domains  to  obtain  a complete  picture,  for  them,  of  an  occupation,  when  their  interests  are  most 
likely  to  be  restricted  to  one  or  a few  domains..  On  the  other  hand,  some  descriptors  within  a 
domain  will  no  doubt  appear  to  be  and  may,  empirically,  turn  out  to  be  highly  related  to  some 
descriptors  in  another  domain—because  of  the  desire  for  within-domain  completeness. 

Each  domain  of  descriptors  has  been  developed  to  apply  across  occupations.  That  is,  the 
level  of  detail  and  definition  was  intended  to  allow  the  sensible  use  of  the  descriptors  across  all 
occupations,  but  still,  when  used  collectively,  be  useful  for  clear  discriminations  between 
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occupations.  The  scales  accompanying  each  occupational  descriptor  (e.g.,  level  of  complexity  or 
importance)  were  developed  to  allow  quantitative  descriptions  of  and  discriminations  between 
occupations. 

Partially  apparent  in  Figure  1-1  is  the  hierarchical  nature  of  the  0*NET  system.  Note  that 
abilities,  occupational  values  and  interests,  and  work  styles  are  grouped  under  worker 
characteristics,  while  generalized  work  activities,  work  context,  and  organizational  context  are 
grouped  under  occupational  requirements.  This  descending  hierarchy  continues  to  several  more 
levels,  as  described  in  the  content  model  report  and  in  later  chapters  in  this  report.  Use  of 
hierarchies  in  this  manner  allows  easier  access  and  understanding  of  the  large  numbers  of 
variables  in  the  content  model;  each  user  can  access  the  information  at  the  appropriate  level. 
However,  this  incurs  the  cost  of  examining  the  appropriateness  and  limits  of  aggregating 
quantitative  data  up  from  the  most  basic  level.  In  the  prototype  project,  all  data  were  collected  at 
the  lowest  levels  of  these  hierarchies  and  initial  investigations  of  data  aggregation  were 
conducted. 

Also  somewhat  implicit  in  Figure  1-1  is  the  relational  nature  of  the  0*NET  system.  In  a 
relatively  mundane,  but  important,  sense,  this  refers  to  the  fact  that  all  the  occupational  descriptor 
data  are  ultimately  housed  in  an  electronic,  relational  database  with  the  various  data  fields  linked 
by  a common  occupational  code.  In  a larger  sense,  it  refers  to  the  notion  that  all  the  various 
domains  are  conceptually  and  theoretically  linked,  as  outlined  in  the  content  model  report,  and 
are  susceptible  to  being  empirically  linked  through  analyses  of  the  quantitative  data.  We  have 
taken  the  first  steps  toward  these  empirical  investigations  in  this  report. 

Perhaps  most  importantly,  but  not  apparent  in  the  figure,  is  the  notion  that  the  0*NET 
provides  a common  language  for  describing  and  thinking  about  occupations.  The  actual 
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definitions  and  rating  scales  that  accompany  each  descriptor  provide  the  basic  elements  of  this 
language,  the  words.  The  theoretical  and  empirical  relationships  between  the  descriptors  within 
each  domain  and  across  the  domains  begin  to  provide  a sort  of  grammar  for  the  occupational 
universe-the  ways  in  which  the  words  fit  together  to  provide  information  about  occupations. 
Accompanying  this  notion  is  the  very  real  point  that  language  is  not  static,  but  constantly  . 
evolving-although  not  at  a rate  that  obviates  the  usefulness  of  the  language. 

The  remainder  of  this  report  describes  the  initial  attempt  to  make  operational  the 
0*NET  content  model-that  is,  to  identify  occupations  upon  which  to  collect  0*NET 
information  in  a systematic  way  and  to  analyze  that  data  to  discover  its  reliability,  its  internal 
structure,  its  ability  to  differentiate  and  to  cluster  occupations,  and  its  usefulness  for  describing 
occupations.  Chapter  2 describes  the  research  methodology,  including  sampling  data  collection 
procedures,  and  the  general  analytic  methods  employed.  The  next  nine  chapters  describe  the 
results  of  the  analyses  of  the  data  for  each  0*NET  domain:  skills,  knowledges,  occupational 
preparation,  generalized  work  activities,  work  context,  organizational  context,  abilities, 
occupational  values,  and  work  styles.  Chapters  12  through  16  cover  topics  that  cut  across  the 
domains,  including  possible  covariates  of  0*NET  rating  data,  cross-domain  analyses,  cluster 
analyses  of  occupations,  use  of  0*NET  data  to  describe  occupations,  and  analyses  and  methods 
of  collecting  occupation-specific  information.  The  final  chapter  presents  some  general 
conclusions  and  recommendations  for  future  research  and  development. 
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Introduction  and  Background 

Our  objective  in  developing  the  taxonomies  of  cross-occupation  descriptions  which  form 
the  content  model  was  to  provide  a basis  for  measuring  the  similarities  and  differences  observed 
among  occupations.  A variety  of  procedures  might  be  used  to  describe  occupations  with  respect 
to  their  status  on  these  variables.  One  might,  for  example,  develop  an  objective,  formal  test 
intended  to  assess  workers’  expressions  of  each  characteristic.  Alternatively,  one  might  ask 
occupational  analysts  to  observe  worker  performance  and  then  rate  the  extent  to  which  each 
variable  was  required  for  effective  performance. 
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In  fact,  ratings  represent  the  technique  most  commonly  used  to  obtain  descriptions  of 
occupation  characteristics.  To  obtain  these  ratings,  definitions  of  the  variables  of  interest  are 
presented  to  people  familiar  with  the  occupation.  Judges  are  then  asked  to  describe  how  the 
variables  manifest  themselves  in  job  performance,  using  rating  scales  which  examine  different 
ways  the  variables  might  be  expressed.  For  example,  one  might  ask  judges  to  describe  how 
frequently  an  activity  is  performed  and  how  important  it  is  to  job  performance. 

This  widespread  use  of  ratings  in  the  description  of  occupation  characteristics  is 
attributable  to  three  characteristics  of  the  approach.  First,  ratings  allow  a great  deal  of  descriptive 
information  to  be  collected  at  a relatively  low  cost.  Second,  ratings  allow  us  to  draw  on  the 
expertise  of  the  people  doing  the  work  in  assessing  occupational  requirements.  Third,  because  a 
variety  of  different  types  of  rating  scales  might  be  developed,  it  is  possible  to  look  at  the  different 
ways  a given  descriptor  variable  manifests  itself  on  people’s  jobs. 

This  final  comment  brings  to  the  fore  a new  question:  exactly  what  type  of  rating  scales 
should  be  used  in  describing  occupations?  A variety  of  different  types  of  rating  scales  have  been 
developed.  Fleishman’s  (1982)  ability  requirement  ratings,  for  example,  are  based  on 
behaviorally  anchored  scales  examining  the  level  of  the  ability  required  for  task  performance.  A 
number  of  other  scales  for  describing  ability  requirements  have  been  suggested,  such  as  the 
importance  of  the  ability  in  accounting  for  observed  performance  differences  or,  alternatively,  its 
impact  on  performance  under  emergency  conditions  (P.  D.  Sackett,  personal  communication, 
1993). 

As  Messick  (1995)  points  out,  selection  of  a particular  rating  scale  should  be  based  on  the 
nature  of  the  variable  at  hand  and  the  type  of  inferences  to  be  drawn.  Thus,  Work  Context 
variables  (e.g..  Noise)  might  be  assessed  in  terms  of  frequency  or  intensity.  Frequency  and 
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intensity  scales,  however,  would  not  be  appropriate  for  assessing  skills  and  abilities.  Thus,  in 
developing  the  0*NET  system,  different  types  of  rating  scales  were  used  to  assess  the  variables 
included  in  different  taxonomies  (Mumford,  Weeks,  Harding,  & Fleishman,  1987).  The  types  of 
rating  scales  to  be  applied  were  selected  based  on:  (1)  the  key  manifestations  of  the  variable  on 
people’s  jobs,  (2)  the  feasibility  of  applying  the  scale  across  occupations,  (3)  the  usefulness  of  the 
descriptive  information  provided,  (4)  the  appropriateness  of  the  scale  for  observational  ratings, 
and  (5)  the  available  evidence  bearing  on  the  reliability  and  validity  of  the  resulting  descriptive 
information  (Friedman  & Harvey,  1986;  Harvey  & Lozada-Larsen,  1988). 

Although  rating  scales  were  necessarily  specific  to  the  particular  types  of  variables  under 
consideration,  certain  general  characteristics  of  these  rating  scales  should  be  noted.  To  begin,  in 
most  cases,  multiple  types  of  ratings  were  obtained  for  a given  set  of  variables.  Multiple  ratings 
were  obtained  primarily  because  a number  of  questions  might  be  asked  about  the  variables 
included  in  a taxonomy  (Cornelius,  Carron,  & Collins,  1979;  Cornelius,  Hakel,  & Sackett,  1979). 
Use  of  multiple  ratings,  however,  also  offered  some  advantages  by  forcing  people  to  carefully 
consider  the  questions  being  asked  and  to  distinguish  among  constructs-a  feature  of  multiple 
rating  tasks  that  contributes  to  the  reliability  and  validity  of  the  resulting  descriptive  information 
(McCormick,  1964).  Second,  in  all  cases  where  level  ratings  were  applicable— abilities  and  skills, 
for  example— level  ratings  were  obtained,  based  on  their  demonstrated  reliability  (Fleishman  & 
Quaintance,  1984;  Fleishman  & Mumford,  1988)  and  the  merits  of  level  ratings  in  addressing 
questions  about  job  demands.  In  accordance  with  the  procedures  recommended  by  Fleishman  and 
Mumford  (1988),  these  level  ratings  were  obtained  using  seven-point  behaviorally  anchored 
rating  scales.  These  anchors  were  developed  using  standard  judgmental  scaling  procedures 
(Peterson,  Mumford,  Borman,  Jeanneret,  & Fleishman,  1995;  Childs  & Whetzel,  1995).  Most 
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other  ratings,  however,  were  obtained  on  standard  four-  or  seven-point  Likert  Scales.  In  this 
regard,  however,  it  is  important  to  note  that,  whenever  possible,  an  attempt  was  made  to  develop 
anchors  that  could  be  scaled  on  an  absolute  rather  than  a relative  continuum.  Absolute  scales 
(“How  much  time  do  you  spend  on  this  activity?”)  were  used  rather  than  relative  scales  (“Do  you 
spend  more  time  on  this  activity  as  compared  to  other  activities?”)  in  order  to  facilitate  cross- 
occupation  comparisons  (Harvey,  1990). 

In  developing  these  rating  scales,  a number  of  other  steps  were  taken  to  help  insure  the  • 
reliability  and  validity  of  the  resulting  descriptive  information.  First,  all  rating  scales  were 
constructed  using  operational,  rather  than  technical,  definitions  written  to  clearly  convey  the 
nature  of  the  variables,  eliminate  jargon,  and  minimize  reading  difficulty.  Second,  these 
operational  definitions  of  the  variables,  along  with  the  associated  rating  scales,  were 
administered  to  some  250  people,  working  on  jobs  ranging  from  construction  worker  to  college 
professor  (Peterson,  Mumford,  Borman,  Jeanneret,  & Fleishman,  1995).  In  this  pilot  study,  after 
respondents  had  used  these  rating  scales  to  describe  their  jobs,  they  were  interviewed  one  or 
more  times.  Respondents’  comments  were  used  to  identify  any  ambiguities  in  rating  instructions, 
variable  labels,  operational  definitions,  and  rating  scale  anchors.  Finally,  several  experienced 
occupational  analysts  were  asked  to  review  the  rating  instructions,  variable  labels,  operational 
definitions,  and  rating  scale  anchors  for  clarity,  readability,  and  appropriateness  for  use  in 
describing  people  s jobs.  The  revisions  resulting  from  workers’  and  analysts’  comments  resulted 
in  revised  scales  and  rating  procedures  that  could  be  expected  to  yield  adequate  reliability  and 
validity,  while  casting  scales  in  a common  language  framework  intended  to  facilitate 
communication  and  minimize  the  burden  placed  on  raters. 
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Figure  2-1  presents  the  rating  scales  developed  for  one  Skill  and  one  Generalized  Work 
Activity.  The  instructions  accompanying  these  rating  scales  are  illustrated  in  Figure  2-2,  which 
presents  the  instructions  developed  for  the  skills  questionnaire.  These  rating  scales  typify  the 
basic  instruments  used  to  assess  similarities  and  differences  among  occupations  with  respect  to 
the  cross-occupation  descriptors. 

In  developing  these  measures,  it  was  necessary  to  address  two  other  procedural  issues:  (1) 
what  source  would  be  used  to  obtain  ratings  and  (2)  what  data  collection  formats  would  be  used 
to  gather  these  ratings?  When  occupation  characteristics  are  to  be  described  using  judgmental 
techniques,  such  as  ratings,  it  is  necessary  to  obtain  descriptive  information  from  people  who 
have  the  expertise  needed  to  provide  accurate,  meaningful  assessments  of  occupational 
requirements  (Cornelius,  Denisi,  & Blencoe,  1984).  Broadly  speaking,  three  types  of  raters  are 
available  who  have  the  background  needed  to  provide  adequate  assessments  of  occupation 
characteristics:  (1)  workers,  or  incumbents,  (2)  supervisors,  and  (3)  occupational  analysts. 
Occupational  analysts’  ratings  are  often  seen  as  the  best  source  of  descriptive  ratings,  in  part 
because  analysts  are  viewed  as  more  objective,  and  also  because  analysts  are  seen  as  having  a 
better  understanding  of  how  the  occupation  at  hand  compares  to  other  occupations  (Henderson, 
1988). 

On  the  other  hand,  a variety  of  available  evidence  indicates  that  analysts  may  not 
necessarily  yield  superior  descriptions,  at  least  under  conditions  where  there  is  no  overt  motive 
for  faking.  In  one  study  along  these  lines,  Fleishman  and  Mumford  (1988)  examined  the  degree 
of  agreement  observed  on  ability  requirement  ratings  obtained  from  incumbents,  supervisors,  and 
occupational  analysts.  They  found  that  in  the  kind  of  high  ability  populations  examined  in  this 
study,  all  three  types  of  raters  provided  virtually  identical  descriptions  of  ability  requirements. 
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The  results  obtained  in  the  Fleishman  and  Mumford  (1988)  study  are  by  no  meanc  unique. 
Peterson,  Owens-Kuntz,  Hoffman,  Arabian,  and  Whetzel  (1990)  had  soldiers,  their  supervisors, 
and  occupational  analysts  assess  the  knowledge,  skill,  ability,  and  work  style  requirements  of  a 
sample  of  army  jobs.  Again,  it  was  found  that  these  three  types  of  judges  displayed  substantial 
agreement. 

Taken  as  a whole,  these  studies  suggest  that  occupation  descriptions  might  be  obtained 
from  any  of  these  three  types  of  judges.  Incumbents,  however,  seemed  best  able  to  provide 
information  across  all  descriptor  domains,  especially  complex,  difficult-to-observe  occupation 
characteristics,  such  as  work  styles  and  organizational  context  variables.  Large  samples  of 
knowledgeable  incumbents  are  available,  which  should  contribute  to  the  reliability  of  the 
resulting  descriptive  system.  This  same  consideration,  of  course,  recommends  the  use  of 
incumbents’,  as  opposed  to  supervisors’,  ratings  since  it  generally  is  easier  to  obtain  sizable 
samples  of  incumbents. 

Although  these  considerations  recommend  the  use  of  incumbents’  ratings,  one  further 
point  should  be  borne  in  mind.  Incumbents’  ratings  essentially  base  description  on  the  smallest 
possible  unit  of  analysis— the  positions  of  individual  workers.  By  basing  description  on  these 
discrete  units,  without  the  imposition  of  an  up  front  structure,  it  becomes  possible  to  formulate  a 
more  flexible  occupational  information  system  than  would  be  the  case  if  analysts’  or  supervisors’ 
ratings  were  used.  Regarding  type  of  rater,  we  were  able  to  take  advantage  of  a related 
Department  of  Labor  project  to  convert  the  Dictionary  of  Occupational  Titles  (DOT;  U.S. 
Department  of  Labor,  1991)  information  to  0*NET  information.  In  this  effort,  occupational 
analysts  used  DOT  task  information  about  occupations  to  rate  occupations  on  0*NET  descriptor 
scales.  This  allowed  the  possibility  of  comparing  the  two  kinds  of  ratings. 
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Not  only  can  ratings  be  obtained  from  a number  of  different  types  of  judges,  ratings  might 
be  obtained  using  a number  of  different  data  collection  techniques.  The  most  common  procedure 
used  to  collect  ratings  is  the  traditional  paper-and-pencil  measure.  Paper-and-pencil 
questionnaires  represent  a flexible  data  collection  technique  that  minimizes  costs  while  ensuring 
that  the  measures  can  be  applied  across  a wide  range  of  job  settings.  Nonetheless,  a number  of 
other  techniques  are  available.  For  example,  computer  assisted  telephone  interview  (CATI)  or 
survey  “fax  back”  techniques  might  at  some  time  represent  viable  alternatives  to  traditional 
paper-and-pencil  surveys.  Rating  questionnaires  might  also  be  administered  using  computer 
assisted  data  collection  techniques,  such  as  DOS-  or  Windows-based  administration  of  the  paper- 
and-pencil  surveys  or  use  of  the  World  Wide  Web  as  a basis  for  survey  administration.  Computer 
assisted  survey  administration,  of  course,  reduces  data  coding  costs,  while  opening  up  the 
possibility  of  applying  new  types  of  unobtrusive  measures  during  data  collection.  On  the  other 
hand,  however,  these  techniques  cannot  readily  be  applied  on  many  lower  level  jobs  lacking  easy 
access  to  the  requisite  technology. 

Based  on  the  flexibility  of  paper-and-pencil  techniques,  and  the  need  to  cover  a wide 
range  of  occupations,  the  paper-and-pencil  approach  was  the  primary  method  used  in 
administering  the  rating  scales  developed  in  the  present  effort.  However,  the  feasibility  of  using 
computer-administered  versions  of  the  paper-and-pencil  surveys  was  assessed  in  a sample  of  ten 
occupations  where  incumbents  were  likely  to  have  access  to  the  requisite  technology. 

These  rating  scales  were  also  developed  for  possible  administration  over  the  World  Wide 
Web.  However,  web-based  administration,  because  it  prohibits  systematic  sampling,  was 
considered  a purely  exploratory  technique. 
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Assessing  Internal  and  External  Validity 

No  matter  how  careful  the  procedures  used  in  scale  development,  these  procedures  cannot 
provide  empirical  evidence  for  the  meaningfulness  of  the  occupation  descriptions  obtained  from 
the  O NET  cross-occupation  descriptors.  Accordingly,  there  is  a need  for  evidence  indicating 
that  meaningful  inferences  about  the  characteristics  of  people’s  jobs  can  be  obtained  from  those 
measures.  Fleishman  and  Mumford  (1991)  proposed  a set  of  tests  that  might  be  used  to  accrue 
evidence  for  the  meaningfulness  of  the  descriptive  data  provided  by  job  analysis  measures, 
drawing  a distinction  between  two  basic  types  of  evidence.  They  argue  that  evidence  bearing  on 
the  meaningfulness  of  cross-occupation  taxonomies  may  be  obtained  by  considering  internal  or 
external  validity  evidence. 

Essentially,  internal  validity  examines  the  relationships  embedded  in  the  taxonomy  as  a 
basis  for  drawing  inferences  with  regard  to  its  meaningfulness.  For  example,  one  might  examine 
the  degree  of  agreement  observed  among  judges  in  describing  an  occupation  or  set  of 
occupations  (Cronbach,  1990;  Fleishman  & Mumford,  1991).  Alternatively,  one  might  use 
techniques  such  as  confirmatory  factor  analysis  to  assess  whether  the  variables  included  in  the 
taxonomy  evidence  a meaningful  pattern  of  relationships.  Although  those  various  internal 
validity  indicators  provide  some  important  evidence  for  the  meaningfulness  of  a descriptive 
system,  it  is  important  to  bear  in  mind  that,  ultimately,  these  taxonomies— and  the  resulting 
measures— are  intended  to  describe  the  similarities  and  differences  among  occupations.  Thus,  an 
essential  piece  of  evidence  that  must  be  provided  in  evaluating  any  taxonomy  used  to  describe 
people  s jobs  is  whether  the  resulting  descriptive  system,  in  fact,  provides  meaningful 
discrimination  of  occupation^. 
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The  kind  of  internal  validity  tests  described  above  examine  inferences  drawn  from  within 
a particular  descriptive  system-for  example,  the  consistency  and  reliability  of  descriptors  within 
a particular  domain,  and  the  intuitive  sensibility  of  the  empirical  relationships  of  descriptors 
within  a domain.  External  validity  assessments  attempt  to  establish  whether  inferences  about 
occupations  divined  from  a given  set  of  descriptors,  in  fact,  can  be  supported  by  considering 
relationships  that  go  beyond,  or  were  not  considered,  in  initial  development  of  the  classification. 
For  example,  some  evidence  for  the  skills  taxonomy  might  be  provided  by  showing  that 
occupations  held  to  require  problem  solving  skills  also  tend  to  involve  similar  generalized  work 
activities,  a separate  0*NET  taxonomic  domain.  In  accruing  external  validation  evidence  for  the 
meaningfulness  of  a taxonomy,  a variety  of  different  types  of  tests  of  this  nature  might  be 
conducted  (Messick,  1989, 1995). 

These  introductory  remarks  set  the  stage  for  the  detailed  consideration  of  the  analyses  of 
the  data  collected  using  the  0*NET  occupational  analysis  instruments  on  a sample  of 
occupations.  The  remainder  of  this  chapter  describes  the  sampling  methods,  data  collection 
procedures,  obtained  response  rates  and  associated  issues,  and  the  general  analyses  that  were 
conducted  to  evaluate  the  internal  and  external  validity  of  the  0*NET  system. 

Sampling 

In  order  to  facilitate  assessment  of  internal  or  external  validity,  data  describing  people’s 
jobs  with  respect  to  the  relevant  descriptors  is  required.  One  cannot  draw  general  conclusions 
about  the  meaningfulness  of  the  resulting  information  unless  data  have  been  obtained  from  an 
adequate,  reasonably  representative  sample  of  occupations.  Accordingly,  substantial  effort  was 
devoted  to  obtaining  a representative  sample  of  occupations  for  use  in  assessing  the  0*NET 
descriptive  system,  employing  a multi-stage  sampling  design.  The  following  section  details  each 
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of  the  three  sampling  stages  Selecting  Occupations,  Identifying  Establishments,  and  Selecting 
Incumbents--for  both  the  baseline  and  feasibility  studies,  and  explores  some  of  the  caveats 
applicable  at  each  stage  of  the  sampling  design.  Figure  2-3  depicts  these  three  major  stages  of 
sampling  for  the  baseline  and  feasibility  studies. 

Selecting  Occupations 

The  framework  used  for  sampling  occupations  was  based  on  the  1984  Occupational 
Employment  Statistics  (OES)  classification,  which  describes  occupations  in  terms  of  a six-digit 
code.  The  first  two  digits  of  this  six-digit  system  describe  the  job  family,  while  the  third  digit 
identifies  occupations  within  this  job  family.  The  occupational  sampling  frame  for  the  baseline 
study  was  created  by  merging  the  1991, 1992,  and  1993  Occupational  Employment  Statistics 
(OES)  data  files  with  the  job  family  codes  from  the  National  Crosswalk  Service  Center’s 
(NCSC)  data  file.  The  merged  OES  data  files  provided  a file  of  occupational  data  with  multiple 
records  by  year  for  a given  occupation.  One  record  was  created  for  each  unique  five-digit 
occupation  code  by  collapsing  multiple  occurrences  of  a given  occupation  code  and  cumulating 
total  employment. 

The  OES  and  NCSC  file  merge  was  largely  successful,  with  only  14  NCSC  occupations 
not  receiving  OES  codes.  The  NCSC  file  then  provided  additional  codes  for  10  of  the  non- 
matching OES  occupations.  These  four  unmatched  occupations  were  removed  from  the  sampling 
frame.  Their  occupation  codes,  titles,  and  total  employment  are  as  follows: 
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Occupation  Code 

Occupation  Title 

Total  Emplovment 

22514 

Drafters 

286,920 

79014 

Gardeners  and  groundskeepers 

127,660 

87705 

Pile  driving  operators 

1,400 

93951 

Engraving  & printing  workers,  hand 

3,850 

One  further  restriction  applies  to  the  occupation  sampling  frame. 

Occupations  unique  to 

Federal  government  entities  are  not  represented  in  the  occupation  sampling  frame.  Therefore, 
employment  counts  for  those  occupations  underestimate  the  total  United  States  employment 
counts  by  the  number  of  Federal  employees  in  those  occupations.  (An  OES  supplement  includes 
data  on  Federal  government  entities,  but  this  data  set  excludes  19  large  Federal  entities  and, 
therefore,  we  considered  this  supplemental  data  set  insufficient  for  use  in  the  occupation 
sampling  frame.) 

The  resulting  sampling  frame  is  presented  in  Figure  2-4.  This  figure  provides  counts  of 
occupations,  total  employment,  and  sample  size  by  two-  and  three-digit  job  family  code.  Each 
cell  of  the  figure  provides  information  for  a particular  three-digit  job  family.  These  cells 
represent  areas  of  analytic  interest  and  were  therefore  used  as  strata  for  the  sample  of 
occupations.  Cells  with  fewer  than  two  hundred  thousand  employees  were  excluded  from  the 
sample  design  (i.e.,  they  were  assigned  sample  sizes  of  zero).  There  was  no  information  in  the 
OES  files  for  some  job  families  (e.g.,  job  family  “104”)  and  a small  amount  of  employment  was 
reported  for  three-digit  job  family  codes  that  did  not  appear  in  the  original  job  family  matrix 
provided  to  us.  Both  types  of  cells  were  excluded  from  the  sample  design. 

There  are  several  reasons  for  the  selection  of  a probability  sample  of  occupations,  using 
probability  proportionate  to  size.  A major  reason  for  using  probability  sampling  was  to  have  an 
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objective  method  of  picking  the  occupations  to  study,  thereby  avoiding  conscious  or  unconscious 
biases  in  selection  choices.  This  type  of  sampling  tends  to  result  in  the  selection  of  larger 
occupations  (in  terms  of  number  of  employees),  thereby  permitting  wide  coverage  of  the  work 
force  with  relatively  few  jobs.  Additionally,  when  probability  proportionate  to  size  is  used  for  the 
first  stage  of  selection  and  a constant  number  of  employees  are  chosen  per  occupation,  the 
sample  produces  close  to  equal  probabilities  of  choosing  employees.  This  was  a desirable 
property  of  the  sample,  even  though  no  national  estimates  are  planned.  Therefore,  we  allocated 
70  occupations  across  the  14  columns  of  the  job  family  matrix  with  the  probability  proportional 
to  the  column  total  employment.  This  simply  means  that  occupations  with  larger  employment 
totals  received  a larger  number  of  job  families  to  sample.  Adjustments  were  made  to  the  column 
sample  sizes  so  that  at  least  one  occupation  could  be  selected  within  each  cell.  For  example,  we 
allocated  two  occupations  to  job  family  column  “10”  to  ensure  selection  of  one  occupation  each 
in  rows  “101”  and  “102”  instead  of  the  0.65  total  occupations  warranted  under  strictly 
proportional  allocation.  We  allocated  the  resulting  column  sample  sizes  across  cells  in  a similar 
manner.  The  final  sample  size  for  each  cell  is  given  by  the  last  entry  in  each  table  cell.  A 
sampling  macro  was  then  used  to  select  the  specified  number  of  occupations  within  each  cell 
with  probability  proportional  to  total  employment.  The  70  occupations  selected  in  this  manner 
represent  approximately  45%  of  the  workforce,  consistent  with  our  goal  in  employing  probability 
sampling. 

Since  employer  demands  are  placing  greater  emphasis  on  computer  and  technical  skills, 
there  was  a need  to  ensure  that  the  prototype  included  occupations  considered  to  be  highly 
technological  in  nature.  A limitation  of  the  OES  sampling  frame  is  that  it  taps  occupations 
proportional  to  the  amount  of  current  employment  and  does  not  necessarily  include  emerging  or 
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highly  technological  occupations.  Thus,  it  was  necessary  to  adopt  an  additional  sampling 
technique  in  order  to  include  the  emerging,  technological  occupations. 

A literature  review  suggested  that  highly  technological  occupations  are  prevalent  in  many 
high  performance  organizations  (Arad,  Schneider,  & Hanson,  1995).  These  organizations 
emphasize  the  importance  of  innovation  and  technology  in  the  workplace.  Employment  within 
high  performance  organizations  tends  to  set  industry  standards  and  the  organizations  are  often 
used  as  benchmarks  for  job  and  market  trends.  Thus,  high  performance  workplaces  seemed  a 
logical  place  to  find  the  types  of  occupations  absent  from  the  sample  selected  using  traditional 
statistical  techniques. 

The  literature  review  indicated  that  many  high  performance  organizations  were 
concentrated  within  a few  industries.  Thus,  we  chose  the  most  commonly  represented  industries 
in  the  literature  and  used  the  concatenated  OES  data  files  from  those  industries  to  select 
additional  occupations.  Occupations  considered  for  selection  were  those  that  were  commonly 
found  within  high  performance  industries  and  were  technological  in  nature.  Ten  occupations  that 
met  those  criteria  were  chosen  to  supplement  the  original  sample  of  70  occupations. 

Taken  together,  the  sample  of  occupations  used  for  initial  development  of  the  0*NET 
system  consisted  of  80  unique  occupations-70  of  which  were  selected  proportional  to  current 
employment  size  using  the  OES  sampling  technique,  and  10  of  which  were  selected  from 
industries  frequently  represented  in  the  high  performance  literature.  Figure  2-5  provides  two-  or 
three  digit  job  family  code,  occupation  code,  occupation  title,  and  total  employment  of  the  80 
occupations  selected  for  inclusion  in  the  baseline  study. 
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Selecting  Organizations 

Once  the  80  occupations  were  approved,  a list  of  establishments  that  were  likely  to 
employ  people  in  those  occupations  was  obtained.  Dun  and  Bradstreet  (D&B)  maintains 
complete  and  up-to-date  files  of  all  establishments  in  the  United  States  and  provides  names, 
addresses,  and  phone  numbers  to  customers.  These  data  files  were  used  to  identify 
establishments— business  units— likely  to  employ  people  on  the  targeted  jobs. 

In  communicating  project  requirements  to  D&B,  the  assumption  was  that  four  standard 
industrial  classification  (SICs)  were  sufficient  to  represent  each  occupation.  Thus,  a total  of  three 
hundred  and  twenty  SICs  were  initially  requested  (4  SICs  x 80  occupations);  however,  because 
some  occupations  mapped  to  fewer  than  four  SICs  in  D&B’s  files,  only  143  SICs  were  actually 
selected.  These  SICs  were  chosen  on  the  basis  of  having  a high  probability  of  employing  people 
in  the  80  occupations  of  interest. 

D&B  was  instructed  to  randomly  select  a sample  of  2,160  establishments  from  their  file. 
They  were  asked  to  stratify  the  sample  across  the  143  three-digit  SICs.  D&B  stratified  the  sample 
across  the  following  four  employment  size  classes:  (a)  5 to  24  employees,  (b)  25  to  99 
employees,  ( c)  100  to  499  employees,  and  (d)  500  or  more  employees.  Establishments  that 
employed  fewer  than  five  people  were  excluded,  as  they  were  not  expected  to  yield  enough 
employees  in  each  sampled  occupation. 

Once  the  sample  was  returned  from  D&B,  it  was  partitioned  into  a baseline  sample  (n  = 
1,240  establishments)  and  a reserve  sample  (n  = 720  establishments).  The  reserve  sample  was 
used  only  when  employer  cooperation  was  less  than  anticipated. 

For  roughly  20  occupations,  these  procedures  failed  to  provide  the  requisite  number  of 
establishments  needed  to  obtain  the  targeted  sample  size.  In  some  cases,  the  problem  was  due  to 
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the  need  for  greater  specificity  in  sampling  occupations  (e.g.,  photographers).  In  other  cases,  it 
was  attributed  to  the  number  of  people  employed  on  the  job  at  a given  establishment  (e.g., 
butchers).  Accordingly,  a two-phase  plan  was  developed  to  address  these  problems.  The  first  • 
phase  involved  extending  the  number  of  establishments  sampled  by  an  additional  400,  but  using 
more  specific  SIC  codes.  The  second  phase  involved  identifying  the  remaining  occupations 
where  a sufficient  number  of  potential  participants,  had  not  been  identified  and  then  identifying 
people  employed  on  these  jobs,  using  alternative  sources  for  identifying  workers,  such  as  trade 
associations,  unions,  and  public  announcements  of  the  project. 

Selecting  Incumbents 

Screening 

Once  establishments  employing  people  in  the  targeted  occupations  had  been  identified, 
attempts  were  made  to  contact  each  of  those  establishments,  solicit  their  cooperation,  and  obtain 
a sample  of  incumbents,  people  currently  working  on  the  targeted  job.  Due  to  the  dynamic  nature 
of  businesses,  the  D&B  sample  contained  a certain  amount  of  out-of-date  or  missing  information. 
Additionally,  response  rates  improve  when  researchers  can  identify,  by  name,  the  appropriate 
respondent  prior  to  an  initial  contact.  For  these  reasons,  telephone  interviewers  contacted  sample 
establishments  prior  to  negotiating  study  participation.  Interviewers  called  each  company  in  the 
sample  to  confirm  the  name,  address,  and  size  of  the  organization,  and  to  obtain  the  name  and 
title  of  the  appropriate  person  to  send  an  advance  letter  explaining  the  study.  (These  points-of 
contact  were  generally  human  resource  personnel.)  The  screening  interview  was  designed  to  be 
answered  by  a receptionist  and  took  about  5 minutes  to  administer.  The  three  day  screening 
resulted  in  a sample  of  1,055  establishments.  One  hundred  and  ten  of  the  original  sample  were 
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ineligible  to  participate,  as  they  employed  fewer  than  five  people.  One  hundred  and  seventy-five 
establishments  were  out  of  business,  duplicates,  or  not  locatable,  and/or  refused  to  participate. 

Establishments  qualified  to  participate  in  the  study  were  mailed  an  advance  letter.  The 
advance  letter  described  the  purpose  of  the  study,  discussed  how  they  could  help,  and  provided  a 
fact  sheet  about  the  0*NET  project  which  was  used  to  support  the  solicitation  for  study 
participation.  Copies  of  the  data  collection  materials  are  contained  in  Appendix  B of  this  report. 
Negotiation  and  Incumbent  Selection 

Perhaps  the  most  important  aspect  of  the  data  collection  effort  involved  negotiating  study 
participation  with  the  employer  sample  in  the  baseline  and  feasibility  (diskette-by-mail)  studies. 
Success  in  negotiating  is  necessary  to  ensure  study  participation  and  endorsement  by  employers. 
Once  employers  agree  to  participate,  they  are  asked  to  provide  organizational  data  and  coordinate 
the  collection  of  occupational  information  from  their  employees.  Negotiations  are  also  necessary 
to  confirm  the  existence  of  the  occupations  that  are  expected  to  be  present  in  each  establishment, 
and  the  number  of  persons  employed  in  each.  The  numbers  gathered  during  negotiations  are  used 
to  sample  employees  in  each  occupation  across  all  participating  establishments.  A script  was 
used  in  negotiating  study  participation  with  organizations. 

Due  to  the  complexity  of  these  negotiations  and  the  high  level  of  expertise  needed. 
Occupational  Analysis  Field  Center  (OAFC)  staff  took  primary  responsibility  for  this  activity. 
The  OAFC  analysts  were  provided  with  training  in  negotiation  procedures.  The  negotiation 
involved  contacting  point-of-contact  personnel  (POCs)  at  each  sampled  establishments  by 
telephone  and  providing  information  about  the  study,  gaining  agreement  to  participate,  and 
making  appointments  for  later  interviews.  Prior  to  negotiation,  a list  of  occupations  likely  to  be 
present  within  each  sampled  establishment  was  generated  based  upon  matching  the  establishment 
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with  its  major  SIC.  No  more  than  10  occupations  were  assigned  to  an  establishment.  During  the 
negotiation,  POCs  were  asked  to  confirm  that  these  occupations  were  present  in  the  organization 
and  provide  the  number  of  incumbents  (employees)  in  each  of  those  occupations.  Following 
negotiations,  all  of  the  information  about  the  number  of  incumbents  within  an  occupation 
received  from  these  organizational  representatives  was  then  entered  into  an  employee  sampling 
database. 

Finally,  the  number  of  employees  to  sample  per  occupation  within  an  establishment  was 
generated  based  upon  targeted  sample  sizes.  Sampling  rates  were  based  upon  overall 
employment  within  the  occupation  studied.  For  example,  occupations  like  secretaries  yielded 
large  numbers  of  employees  in  the  sampling  database.  These  large  numbers  led  to  low  sampling 
rates.  Low  sampling  rates  mean  that  the  sample  of  secretaries  was  spread  across  many 
establishments,  with  any  one  establishment  only  asked  for  a small  number  of  employees  in  that 
occupation. 

On  the  other  hand,  a different  outcome  occurred  for  occupations  such  as  librarians,  which 
yielded  smaller  counts  of  employees  in  the  sampling  database.  Occupations  like  this  resulted  in 
higher  sampling  rates.  This  means  that  the  sample  of  librarians,  in  this  example,  was  spread 
across  fewer  establishments,  thereby  requiring  more  employees  to  be  sampled  within  each 
establishment. 

Organizational  Representative  Interview 

Following  the  negotiation  phase,  the  organizational  representatives  (POCs)  of 
organizations  that  agreed  to  participate  were  contacted  again  to  complete  a 40-  to  45-minute 
computer  assisted  telephone  interview  (CATI)  designed  to  be  conducted  with  the  POC.  The 
protocol  for  the  CAT!  is  contained  in  Appendix  A to  this  report. 
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Since  the  organizational  representative  interview  was  originally  developed  for  paper-and 
pencil  administration,  it  was  necessary  to  revise  it  for  CATI  administration.  A team  of  industrial 
psychologists  worked  together  to  shorten  questions  and  simplify  them  as  much  as  possible 
without  destroying  meaning.  The  revised  measure  was  pilot  tested,  via  telephone,  on  three 
volunteer  respondents.  The  respondents  were  human  resource  representatives  with 
responsibilities  similar  to  those  of  eventual  study  respondents.  Respondents  agreed  in  advance  to 
participate  in  the  pilot  interview,  but  were  not  given  any  information  about  the  types  of  questions' 
they  would  be  asked. 

The  pilot  test  revealed  that  the  instrument  was  too  long  overall  when  used  in  interview 
form.  Furthermore,  some  question  stems  were  too  long  as  were  some  of  the  response  choice 
alternatives.  Therefore,  many  of  the  examples  and  definitions  from  the  body  of  the  questionnaire 
were  eliminated.  Care  was  taken  to  leave  examples  and  definitions  intact  if  their  elimination 
would  alter  the  construct  being  measured.  The  examples  and  definitions  that  were  removed  were 
transferred  into  a Question-by-Question  Specification  Guide.  This  guide  was  available  to 
interviewers  so  that  they  could  answer  any  questions  respondents  may  have  in  a standardized 
manner.  In  addition,  several  questions  were  dropped  because  they  duplicated  information 
ascertained  elsewhere.  Following  revisions,  the  interview  time  was  reduced  from  65  to  40 
minutes.  Following  these  revisions,  the  organizational  representative  interview  was  programmed 
using  Cheshire,  Westaf  s proprietary  CATI  system. 

At  the  conclusion  of  the  organizational  survey  interview,  the  POC  was  told  which 
occupation(s)  at  his/her  establishment  were  recommended  for  sampling,  and  the  number  of 
incumbents  needed  in  each  occupation.  During  this  discussion,  the  POC’s  responsibilities  in 
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collecting  job  analysis  questionnaires  from  his  or  her  organization’s  employees  was  reiterated. 
Interviews  with  POCs  occurred  from  October  1995  through  January  1996. 

Following  completion  of  the  CATI  interview,  the  POC  was  sent  a package  containing:  (a) 
a letter  thanking  the  POC  for  participating,  (b)  instructions  for  implementing  the  project,  (c)  a list 
containing  the  number  of  incumbents  needed  to  complete  job  analysis  questionnaires  in  each  of 
the  selected  occupations,  (d)  job  analysis  questionnaire  packets—one  for  each  incumbent,  and  (e) 
materials  that  could  be  distributed  to  explain  the  study  to  managers  and  incumbents. 

The  POC  was  asked  to  randomly  select  incumbents  in  the  designated  occupations  from 
among  those  who  had  been  employed  on  the  job  for  six  months  or  more,  and  to  coordinate  the 
distribution  and  collection  of  job  analysis  questionnaires  from  participating  incumbents. 

The  selected  incumbents  received,  from  the  POCs,  a packet  containing:  (a)  a letter 
introducing  the  study,  (b)  instructions  for  completing  questionnaires,  ( c)  supporting  materials 
and  information,  (d)  a background  information  questionnaire,  (e)  an  occupation-specific  task 
questionnaire,  and  (f)  two  to  four  general  across-occupation  questionnaires. 

Incumbents  were  asked  to  complete  the  questionnaires  at  work  and  return  them  in  a 
sealed  envelope  to  the  POC.  The  POC  then  forwarded  these  packets  for  processing.  Follow-up 
telephone  calls  were  conducted  to  prompt  non-responding  employers.  The  purpose  of  the  follow- 
up was  to  answer  POC  questions  and  to  encourage  study  participation.  An  interview  form  was 
used  in  carrying  out  the  telephone  follow  up  calls.  One  follow-up  phone  call  was  made 
approximately  every  two  weeks  for  the  following  six  weeks  after  the  POC  had  received  his  or  her 


package  of  questionnaires. 
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Data  Collection  Procedures 
Measure  Administration 

To  this  point  we  have  examined  sampling  procedures.  Although  this  background  is 
needed  to  understand  the  nature  of  the  sample,  it  does  not  provide  information  about  how  the 
measures  were  administered  to  sample  members. 

The  sampling  procedures  used  in  initial  development  of  the  0*NET  occupational 
information  system  were  designed  to  provide  30  incumbents  on  each  measure  for  a given  job. 
This  targeted  cell  size  was  selected  based  on  two  considerations.  First,  samples  of  20  to  30 
individuals  typically  yield  stable  means  and  standard  deviations  (Winer,  1971)~a  point  of  some 
importance  if  these  measures  are  to  be  used  in  describing  people’s  jobs.  Second,  prior  research 
by  Fleishman  and  his  colleagues  (e.g.,  Fleishman  & Mumford,  1988;  Fleishman  & Quaintance, 
1984)  and  others  (e.g..  Hunter,  1980)  indicates  that  15  to  30  incumbents  are  typically  sufficient  to 
obtain  adequate  interrater  agreement  coefficients,  given  the  type  of  measures  being  used  to 
describe  people’s  jobs. 

To  obtain  30  incumbents  assessments  of  their  jobs  on  each  measure,  however,  it  was 
necessary  to  use  sampling  procedures  that  provided  roughly  100  incumbents  for  each  of  the 
targeted  occupations.  The  need  for  such  a large  number  of  incumbents  was  based  on  the  structure 
of  the  content  model  and  its  implications  for  data  collection.  More  specifically,  the  content  model 
included  a number  of  different  descriptor  domains  (e.g.,  skills  and  knowledges).  Furthermore, 
within  any  given  domain-skills,  for  example-it  was  expected  that  a relatively  large  number  of 
descriptors  would  be  required,  roughly  30  or  40.  Because  organizations  could  not  be  expected  to 
provide  more  than  90  minutes  of  an  incumbent’s  time,  not  all  measures  could  be  administered  to 
all  incumbents. 
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To  minimize  testing  time  and  ensure  that  the  relationships  among  measures  could  be 
examined,  we  used  a rotation  design  in  administering  the  rating  scales.  In  this  rotation  design,  the 
measures  developed  for  a given  description  domain-skills,  for  example- were  treated  as  a 
distinct  questionnaire.  The  questionnaires  were  grouped  into  packets,  where  each  packet 
contained  2 or  3 distinct  questionnaires,  accompanied  by  background  material  and  a short  task 
inventory,  and  took  60  to  90  minutes  to  complete.  In  all,  15  different  packets  were  assembled. 
These  packets  were  rotated  across  five  administration  sets,  such  that  each  questionnaire  was 
paired  at  least  once  with  every  other  questionnaire.  Figure  2-6  presents  this  rotation  design, 
noting  both  the  time  needed  to  complete  each  questionnaire  and  the  time  needed  to  complete  all 
questionnaires  included  in  the  packet. 

This  design  required  100  incumbents  on  each  job  in  order  to  obtain  33  incumbents  who 
responded  to  each  questionnaire  and  a smaller  number  (minimum  = 6)  of  incumbents  who 
responded  to  any  given  pair  of  questionnaires.  Accordingly,  this  design  required  8,000 
incumbents  overall  across  the  80  occupations  considered  in  development  of  the  0*NET  content 
model.  An  additional  1,000  incumbents  were  required  as  part  of  the  methods  feasibility  study,  to 
be  given  computer-administered  versions  of  the  questionnaires. 

With  regard  to  this  design,  four  other  points  should  be  mentioned.  First,  questionnaires 
were  rotated  across  packets  to  minimize  potential  order  and  priming  effects  (Morgeson  & 
Campion,  1996).  However,  within  a questionnaire,  order  was  fixed  to  maintain  the  structure  of 
the  taxonomies  and  reduce  burdens  placed  on  the  incumbents.  Second,  all  packets  began  with  an 
introduction  describing  the  study,  stressing  that  the  data  were  being  gathered  as  part  of  a national 
research  study.  Third,  following  this  introductory  material,  all  incumbents  were  asked  to 
complete  a short  background  information  form,  where  they  were  asked  to  indicate  age,  sex. 
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educational  level,  and  years  of  experience.  Fourth,  once  they  had  completed  the  various 
questionnaires  included  in  their  packet,  all  incumbents  were  asked  to  rate  the  frequency  with 
which  they  performed  and  the  importance  of  certain  tasks  on  their  jobs,  identified  by 
occupational  analysts  using  available  DOT  data  for  the  targeted  occupations. 

Due  to  the  nature  of  this  design  and  the  measures  in  use,  separate  instructions  were 
formulated  for  each  questionnaire.  Incumbents  were  asked  to  read  these  instructions  before 
starting  work  on  a questionnaire.  These  instructions,  although  they  necessarily  differed  across 
questionnaires,  followed  a common  foimat.  They  began  by  describing  the  general  nature  of  the 
rating  task,  asking  incumbents  to  describe  their  current  job  using  the  provided  rating  scales.  Next 
they  described  in  some  detail  the  nature  of  each  rating  scale,  carefully  distinguishing  between 
different  ratings  (e.g.,  level  and  importance)  being  collected  for  the  variables  included  in  a given 
questionnaire.  Finally,  an  example  of  the  ratings  was  provided.  Figure  2-2  provides  an 
illustration  of  these  instructions,  in  this  case,  the  instructions  accompanying  the  skills 
questionnaire. 

Comparative  Studies 
Methods  Comparison  Study 

To  examine  the  feasibility  of  collecting  descriptive  information  on  content  model 
variables  using  other,  alternative  data  collection  techniques,  a method  comparison  study  was 
conducted.  The  alternative  data  collection  methods  used  in  this  comparison  included  mail-out  of 
diskettes  containing  computer-administered  versions  of  the  questionnaires. 

To  conduct  this  methods  comparison  study,  10  occupations  in  which  workers  were  likely 
to  have  access  to  personal  computers  (e.g.,  chemical  engineers)  were  selected  from  among  those 
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included  in  the  baseline'  study.  For  each  occupation,  70  incumbents  were  asked  to  complete  the 
questionnaires  using  a DOS-based  version  of  the  questionnaires. 

In  all,  200  establishments  were  needed  to  complete  this  portion  of  the  study.  The  selected 
establishments,  from  which  the  sample  members  were  drawn,  did  not  overlap  with  the 
establishments  in  the  baseline  paper-and-pencil  study.  The  screening  administration  procedures 
were  the  same  as  those  used  in  the  baseline  study,  with  the  addition  of  one  question.  This 
question  asked  which  computer  operating  system  the  company  used  (e.g.,  DOS,  OS/2,  Unix).  Of 
the  200  companies  initially  included  in  the  feasibility  study,  177  (89%)  remained  eligible  for  the 
study  based  upon  the  results  of  screening.  Of  those  companies  which  became  ineligible,  several 
had  fewer  than  5 employees  and  12  others  were  out  of  business  or  not  locatable.  The  advance 
letter  was  sent  at  the  same  time  as  the  baseline  study,  and  negotiations  resulted  in  150 
participating  companies.  Following  negotiations,  the  POC  within  each  organization  was  asked  to 
complete  a CATI  questionnaire  on  organization-  and  location-specific  information.  The  POCs 
were  asked  to  distribute  and  collect  diskettes  from  employees  in  each  of  the  selected  occupations. 
The  mail-out  of  the  diskettes  began  in  late  November  and  continued  through  January  of  1996. 

Specific  procedural  differences  between  the  baseline  and  feasibility  studies  are  outlined 

below: 

1 . Screener.  The  screener  asked  a few  additional  questions  about  the  type  of 
computers  and  operating  systems  used  in  the  workplace.  These  questions  were  designed  to 
determine  which  organizations  would  be  eligible  to  participate  in  the  method  feasibility  study. 

2.  Advance  Letter.  None. 


3. 


Negotiations.  None. 


2-24  Peterson,  Mumford,  Levin,  Green,  & Waksberg 


4.  Organizational  Representative  Interview.  Text  concerning  the  mode  of  data 
collection  for  employees  differed.  This  text  automatically  appeared  in  the  C ATI  system  for  cases 
in  our  method  feasibility  sample.  POCs  in  this  study  were  also  asked  about  the  types  of 
computers  used  and  the  size  and  type  of  diskette  drives  available. 

5.  Mailout.  Additional  instructions  for  running  the  computer  program  were  included 
in  the  employee  letter  for  the  feasibility  study. 

6.  Follow-up.  None. 

Programming  diskette-by-mail  questionnaires  using  a DOS-based  application.1  The  DOS 
version  of  the  questionnaires  was  developed  to  mirror  the  format  of  the  paper-and-pencil 
instrument  as  much  as  possible.  However,  the  computer  screen  size  allows  only  twenty-two  lines 
of  text  at  one  time,  while  an  average  of  sixty-six  lines  of  text  can  appear  on  a printed  page.  This 
limitation  was  solved  by  keeping  individual  questions  intact,  but  breaking  question  groups  across 
several  screens. 

All  nine  questionnaires,  the  background  information  section,  and  occupation-specific 
sections  were  programmed  onto  one  executable  computer  diskette.  However,  as  in  the  baseline 
study,  a modular  design  was  also  implemented.  The  computer  program  automatically  identified 
which  combination  of  the  nine  questionnaires  modules  and  which  occupation-specific  task  items 
was  to  be  presented  for  each  employee.  This  was  determined  by  a unique  employee  identification 


1 A Windows™  version  of  the  diskette-by-mail  questionnaire  was  completed  in  August  of  1996.  The  decision  to  add 
a Wirdows-based  computer  program  offers  several  advantages  over  the  DOS-based  program.  This  format  provides 
a co.  "fill  layout  and  has  more  flexibility  regarding  font,  size,  and  placement  of  objects  and  characters.  Windows- 
baseo  programs  also  afford  more  flexibility  and  design  freedom  than  other  software.  Documentation  of  the  DOS 
and  Windows  versions  of  the  questionnaires  is  contained  in  the  companion  report,  Q*NET:  An  Information  System 
for  the  Workplace.  Designing  an  Electronic  Infrastructure  (Rose,  Hesse,  Silver,  & Dumas,  1996). 
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number  which  was  linked  to  the  occupation  and  the  questionnaire  module  as  per  the  rotation 
design. 

The  DOS  program  was  developed  using  the  Clipper  5.2  programming  language.  The 
program  was  designed  to  run  on  an  AT-level  machine  with  DOS  version  3.31  or  higher  with  or 
without  a color  monitor.  All  employees  received  the  program  on  3-1/2”  high  density  diskettes, 
unless  they  requested  5-1/4  inch  diskettes.  Some  of  the  features  of  the  DOS  system  include: 

• Instructions  at  the  beginning  of  the  instrument  and  a “thank  you”  at  the  end; 

• All  questionnaire  items; 

• Pop-up  windows  to  display  descriptions  of  “HIGH”  and  “LOW”  values  for  some 
questions; 

• A file  for  storing  respondent  answers,  reducing  the  need  for  editing  and  coding; 

• Navigation  keys  to  move  backward  and  forward  within  the  program; 

• An  exit/re-entry  facility  that  allows  respondents  to  return  to  a partially  completed 
questionnaire;  and 

• Color  for  emphasis  in  distinguishing  between  the  question  text  and  the  text  associated 
with  valid  responses. 

Rater  Comparison  Study 

In  addition  to  the  method  comparison  study,  a rater  comparison  study  was  also  conducted. 
The  intent  of  this  study  was  to  determine  whether  the  descriptions  obtained  from  incumbents, 
people  working  on  the  job,  were  similar  to  the  descriptions  that  were  obtained  from  occupational 
analysts.  Occupational  analysts  providing  these  ratings  were  occupational  analysts  employed  by 
the  OAFCs  and  Industrial/Organizational  psychology  graduate  students.  The  analysts  rated  1,122 
occupational  units,  where  occupational  units  represented  a taxonomy  of  jobs  with  a level  of 
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specificity  in  between  the  more  general  Occupational  Employment  Statistics  (OES)  taxonomy 
and  the  more  specific  Dictionary  of  Occupational  Titles  (DOT)  taxonomy.  Some  of  the  OES 
categories  are  subdivided  and  many  of  the  DOT  categories  are  combined.  The  occupational 
analysts  rated  the  following  0*NET  Content  Model  categories:  basic  and  cross-functional  skills, 
generalized  work  activities  (GW As),  abilities,  work  context,  and  knowledges.  It  is  important  to 
note  that,  although  most  of  the  rating  scales  used  by  the  occupational  analysts  were  identical  to 
those  used  by  job  incumbents,  some  were  not.  In  particular,  the  scale  used  by  occupational 
analysts  to  rate  the  frequency  (in  the  generalized  work  activities  and  work  contexts  domains) 
differed  from  those  used  by  job  incumbents.  The  scales  differed  in  two  ways:  (1)  the  scale  used 
by  the  analysts  had  five  scale  points,  while  the  scales  used  by  the  job  incumbents  varied  from 
five  to  eight  points,  and  (2)  the  scale  anchors  on  the  analysts’  version  did  not  reference  particular 
time  intervals—for  example,  one  anchor  on  the  analysts’  scale  was  “sometimes,”  while  a 
comparable  anchor  on  the  incumbents’  scale  was  “more  than  once  a month.”  In  addition,  analysts 
completed  only  a subset  of  the  work  context  ratings,  and  did  not  rate  job  entry  requirements  for 
skills  or  job  specialty  requirements  for  knowledges. 

For  all  1,122  occupational  units,  each  unit  was  rated  by  at  least  five  raters  independently 
to  minimize  the  effects  of  rater  error.  During  a rating  cycle,  each  rater  rated  a set  of  125 
occupational  units  on  one  0*NET  Content  Model  category  for  level,  importance,  and  frequency, 
when  applicable.  The  80  occupations  targeted  in  the  initial  0*NET  data  collection  are  a subset  of 
the  1,122  occupational  units. 

Receipt  Control 

Since  the  study  design  is  complex,  an  extensive  receipt  control  system  and  information 
database  was  developed  to  track  the  many  phases  of  data  collection.  The  receipt  control  system 
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generated  mailing  labels,  respondent  information  sheets,  individualized  lists  of  occupations  and 
employment  within  those  occupations,  and  individualized  text  used  in  the  CATI  instrument. 
Three  separate  databases  were  constructed  to  include:  the  receipt  control  database:  the  sampling 
database,  and  the  CATI  database. 

The  receipt  control  database  was  the  central  repository  of  information.  It  was  periodically 
updated  with  information  from  the  other  two  databases,  as  well  as  files  generated  from  the  results 
of  the  screener,  study  negotiation,  and  mailout  process. 

The  receipt  control  system  was  built  around  a database  with  a two  tier  file  structure.  File 
one  contained  establishment  level  information.  File  two  contained  respondent  level  information. 
The  establishment  level  file  structure  contained  the  following  information: 

• An  establishment  ID  that  linked  to  the  sampling  database  and  the  respondent  level  file; 

• SIC; 

• A variable  to  indicate  if  the  organization  belonged  to  the  baseline  sample,  the  feasibility 

sample  or  the  reserve  sample; 

• The  organization’s  name  and  address  information  from  D&B,  the  screener  and  the 
negotiation; 

• The  POC’s  name  and  title  from  the  screener; 

• A list  of  likely  occupations  in  each  organization  obtained  from  the  sampling  database; 

• The  list  of  occupations  confirmed  by  the  POC  during  negotiation; 

• The  number  of  incumbents  in  each  occupation  provided  by  the  POC  during  negotiation; 

• Time  and  date  of  the  Organizational  Representative  CATI  scheduled  during  negotiation; 

• The  number  of  employees  sampled  in  each  confirmed  occupation,  derived  from  the 
sampling  database; 
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• Result  codes  and  fielding  dates  for  each  step  in  the  data  collection;  and 

• A trigger  variable  that  signaled  the  next  activity  to  be  performed. 

The  respondent  level  file  contained  the  questionnaire  packet  number,  questionnaire  type, 
and  receipt  control  information. 

The  sampling  database  was  used  for  sampling  occupations  within  establishments  and  for 
identifying  the  number  of  employees  who  were  sampled  in  each  occupation  at  each 
establishment. 

The  purpose  of  the  CATI  database  was  to  link  the  information  gathered  during  the 
organizational  representative  interview  to  the  receipt  control  database.  This  database  directly 
stored  sampling  and  receipt  control  information  obtained  during  the  interview. 

Managing  the  Databases 

Updated  information  was  entered  into  the  databases  during  all  phases  of  the  fielding 
period.  During  the  entire  fielding  period  a nightly  program  was  run,  which  automatically  applied 
files  that  were  updated  that  day.  Applied  updates  could  set  off  switches,  signaling  other  actions  to 
be  taken  by  the  computer.  This  update  process  enabled  many  database  management  functions  to 
take  place  automatically. 


Response  Rates 

Due  to  the  many  stages  of  the  study  design,  response  rates  are  described  below  according 
to  each  of  the  steps  in  the  Study  design.  Results  from  the  baseline  study  are  presented  first, 
followed  by  the  response  rate  results  for  the  feasibility  study. 
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Baseline  Study  Response  Rates 

Initially,  1,240  establishments  were  screened  for  a POC.  Of  the  1,240  establishments, 
1,054  (85%)  were  eligible  to  participate  in  the  study  (see  Figure  2-7).  As  noted  previously,  those 
ineligible  to  participate  included  establishments  that  were  out  of  business,  duplicates  in  the  data 
file,  and  those  having  fewer  than  5 employees.  The  percentage  of  ineligible  organizations 
obtained  from  the  D&B  file  is  typical  of  that  found,  during  the  initial  screening  of  organisations 
for  an  establishment  survey. 

Based  upon  this  initial  screening,  1,054  establishments  were  identified  for  negotiations. 
During  this  phase  of  the  project,  an  additional  80  establishments  (7%)  indicated  that  they  did  not 
have  employees  in  the  jobs  we  were  studying.  During  negotiations,  218  establishments  (21%) 
refused  to  participate  in  the  study  for  a variety  of  reasons  including  time  constraints,  no  interest 
in  the  study,  and  company  policy  regarding  no  participation  in  outside  studies.  Overall,  756 
establishments  (72%)  agreed  during  the  negotiations  to  participate  in  all  aspects  of  the  study 
design. 

The  next  stage  of  the  study  involved  administration  of  the  organizational  interview. 
During  this  stage  an  additional  92  establishments  (12%)  refused  to  participate  even  though  they 
had  initially  agreed  during  project  negotiations.  Organizational  interviews  were  conducted  with 
661  organizational  representatives,  for  a response  rate  of  88%  at  this  stage. 

The  next  stage  of  the  project  involved  mailing  a number  of  incumbent  questionnaires  to 
each  POC  for  distribution  within  the  establishment.  Of  the  661  POCs  who  completed  the 
organizational  interview,  181  returned  at  least  one  incumbent  survey.  This  resulted  in  a response 
rate  at  the  employer  level  of  27%.  During  the  fielding  of  the  incumbent  survey,  it  became 
apparent  early  on  that  POCs  were  not  returning  questionnaires  at  the  rate  anticipated.  In  fact. 
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once  employers  received  the  incumbent  packages,  an  additional  174  establishments  (26%) 
refused  to  participate. 

Identifying  Response  Rate  Problems  in  the  Baseline  Study 

A number  of  actions  were  taken  to  identify  reasons  for  the  low  incumbent  response  rates 
and  to  increase  POC  participation  in  the  distribution  and  collection  of  incumbent  questionnaires. 
First,  follow-up  calls  were  made  to  POCs  who  had  agreed  to  participate,  but  whose  incumbents 
did  not  return  questionnaires.  These  calls  were  intended  to  encourage  POCs  to  administer  the 
surveys.  Second,  a systematic  effort  was  made  to  encourage  POCs  to  distribute  incumbent 
questionnaires  by  making  follow-up  calls  describing  the  importance  of  the  study,  its  benefits,  and 
the  fact  that  the  time  to  complete  the  survey  for  each  incumbent  was  under  an  hour.  Third,  the 
initial  fielding  period  was  extended  from  6 weeks  to  3 months.  Fourth,  a reminder  letter  was 
prepared  and  faxed  to  the  POC  further  encouraging  participation  following  the  final  follow-up 
telephone  call. 

All  of  the  attempts  to  increase  participation  described  above  did  not  significantly  impact 
POCs’  willingness  to  distribute  questionnaires  to  incumbents.  The  high  rate  of  refusal  at  this 
stage  in  the  study  was  not  anticipated  because  POCs  were  carefully  informed  at  earlier  stages 
about  the  time  needed  for  incumbents  to  complete  the  questionnaires  and  the  majority  of  POCs 
agreed  to  participate.  Consequently,  it  appeared  that  the  primary  factor  contributing  to  slow 
response  rates  was  the  POC’s  unwillingness  to  distribute  incumbent  questionnaires—a  gatekeeper 
effect. 

A careful  examination  of  the  baseline  response  rates  support  this  contention.  Results 
presented  in  Figure  2-7  (in  “MAILOUT:  EMPLOYEES  (employees  who  have  returned  data)”) 
show  that  participating  establishments  had  an  incumbent  response  rate  of  60%,  which  was  44% 
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higher  than  the  incumbent  response  rate  obtained  overall.  This  finding  indicates  that  when 
employers  actually  distributed  questionnaires  to  incumbents  in  their  establishment,  participation 
rates  were  quite  satisfactory. 

In  order  to  gain  a better  understanding  of  this  gatekeeper  problem,  phone  calls  were  made 
to  nonresponding  establishments.  The  comments  provided  by  POCs  fall  into  the  following 
categories: 

• Excessive  POC  Burden.  This  perceived  burden  focused  on  the  time  required  of  POCs  to 
sample  incumbents  in  the  selected  occupations  as  well  as  the  time  needed  to  distribute 
and  collect  large  numbers  of  surveys  from  incumbents  (on  average  employers  were  asked 
to  distribute  questionnaires  to  25  incumbents).  In  addition,  many  POCs  who  completed 
the  telephone-administered  organizational  survey  felt  they  had  spent  sufficient  time  on 
this  effort  (about  one  hour  on  the  phone)  and  were  unwilling  to  continue  their 
participation. 

• Time  Required  for  the  Incumbent.  POCs  expressed  strong  reservations  regarding  the  time 
required  of  incumbents  to  complete  the  questionnaires.  Those  who  refused  to  participate 
were  unwilling  to  provide  work  time  to  complete  a questionnaire  taking  60-90  minutes. 
Many  did  not  offer  the  incumbent  the  choice  of  completing  the  questionnaire  at  home. 

• Short  Fielding  Period.  Several  POCs  did  not  have  the  time  to  distribute  the  questionnaires 
over  the  designated  period  of  data  collection  to  incumbents.  If  the  fielding  period  were 
extended,  some  said  they  would  consider  participating. 

• No  Incentive  for  Participation.  Several  POCs  wanted  more  immediate  “payback”  for 
participation.  Suggestions  such  as  financial  compensation  and  use  of  the  0*NET  database 
were  among  the  most  common  incentives  discussed. 
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* Questionnaire  Difficulty.  Some  POCs  felt  the  questionnaire  was  too  difficult  for 
incumbents  to  respond  to  without  assistance.  Language  problems  also  occurred  within 
certain  occupations  which  made  it  impossible  for  the  incumbent  to  complete  the 
questionnaire. 

Pilot  test  to  improve  response  rates.  Based  on  the  above  findings,  a small  pilot  test,  called 
Phase  2,  was  conducted  incorporating  the  feedback  received  from  POCs.  A sample  of  175 
establishments  were  selected  from  the  reserve  sample  for  the  pilot  test.  Two  occupations  likely 
present  in  these  establishments  were  studied— general  managers  and  secretaries.  Specifically,  the 
study  design  incorporated  the  following  features  designed  to  reduce  the  burden  required  from 
sampled  establishments: 

* Reduction  in  Time.  The  questionnaire  modules  were  changed  so  that  each  incumbent 
received  fewer  questionnaires.  This  revision  resulted  in  the  time  for  completion  changing 
from  60-90  minutes  to  30  minutes. 

* Fewer  Incumbents  Sampled  within  an  Establishment.  POCs  were  asked  to  administer  no 
more  than  five  questionnaires  to  incumbents. 

* No  Formal  Sampling  of  Incumbents  within  Establishments  While  procedures  for 
selecting  a representative  sample  of  incumbents  was  described  (e.g.,  range  of  experience, 
range  of  ability),  POCs  were  not  asked  to  follow  formalized  sampling  procedures. 

Change  in  the  Order  of  the  Organizational  Interview.  The  organizational  interview  was 
administered  to  some  of  the  sample  used  in  this  pilot  test.  However,  when  the  survey  was 
administered,  it  was  given  at  the  end  (rather  than  the  beginning)  of  the  data  collection 
process.  That  is,  POCs  were  first  asked  to  distribute  and  collect  incumbent 
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questionnaires.  Then,  a small  sample  of  the  POCs  were  asked  to  participate  in  the 
organizational  interview. 

Results  From  the  Pilot  Test 

Following  screening,  a total  of  169  out  of  175  establishments  (97%)  reached  the 
negotiation  phase  (see  Figure  2-8).  A total  of  1 1 establishments  (17%)  refused  participation  and 
23  establishments  (14%)  were  ineligible  (e.g.,  did  not  have  incumbents  in  the  chosen  occupations 
or  they  had  fewer  than  5 incumbents).  At  the  employer  stage,  65  employers  (51%)  responded  by 
sending  back  incumbent  surveys,  10  (7%)  refused  to  participate,  and  52  (39%)  neither  directly 
refused  nor  sent  back  surveys.  Of  the  questionnaires  mailed,  the  incumbent  response  rate  was 
41%.  Finally,  in  establishments  that  returned  any  data  at  all,  85%  of  the  selected  incumbents 
returned  packets. 

A comparison  of  Figures  2-7  and  2-8  shows  that  the  procedures  used  in  this  pilot  test 
resulted  in  much  higher  incumbent  response  rates  than  those  achieved  in  the  baseline  study. 
Reducing  burden  for  employers  appeared  to  impact  on  their  willingness  to  distribute  and  collect 
questionnaires  from  incumbents.  Clearly,  the  revised  procedures  implemented  in  this  test  should 
be  incorporated  into  any  future  data  collection. 

Other  Procedural  Recommendations 

In  addition  to  some  of  the  methods  and  procedures  examined  in  the  pilot  test  described 
above,  other  notable  lessons  were  learned  from  the  baseline  data  collection.  First,  the  type  of 
establishment  sampling  conducted  relied  on  using  broad  standard  industrial  codes  in  the 
matching  of  establishments  with  occupations.  This  procedure  made  it  difficult  to  identify  a 
sufficient  number  of  establishments  employing  individuals  in  smaller,  less  populous  occupations 
(e.g.,  librarians)  or  occupations  where  many  individuals  are  self-employed  (e.g.,  musicians).  This 
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finding  has  two  noteworthy  implications.  First,  more  detailed  SIC  matching  should  be  used  when 
initially  drawing  the  sample  of  establishments;  second,  in  the  case  of  some  occupations  that  are 
not  typically  found  in  establishments,  it  may  be  better  to  use  trade  association  or  guild  listings  of 
members  as  a basis  for  identifying  individuals  in  particular  occupations. 

Feasibility  Study  Response  Rates 

As  previously  discussed,  the  feasibility  study  examined  the  possibility  of  using  computer- 
based  questionnaire  administration  in  place  of  the  more  traditional  paper  and  pencil  method.  The 
design  of  the  feasibility  study  was  similar  to  that  of  the  baseline  study,  with  the  exception  that  the 
packet  design  used  mirrored  that  of  the  pilot  test.  Consequently,  in  this  study,  incumbents 
received  fewer  questionnaires  than  in  the  baseline  study,  in  order  to  reduce  the  time  required  for 
survey  completion. 

A total  of  200  establishments  were  initially  screened  in  the  feasibility  study.  Of  the  200 
cases,  177  (89%)  were  identified  as  eligible  to  participate.  Establishments  were  identified  as 
ineligible  primarily  because  they  did  not  have  computers  or  were  out-of-business.  Following 
negotiations,  98  establishments  (55%)  agreed  to  participate  in  the  organizational  interview  and 
employee  survey  distribution  and  collection  process.  During  negotiations,  an  additional  54 
establishments  (3 1%)  were  identified  as  ineligible  to  participate,  either  because  they  did  not  have 
DOS  or  Windows  based  PCs  available,  or  because  they  did  not  have  individuals  employed  in  the 
occupations  being  studied.  A total  of  25  establishments  (14%)  refused  to  participate  during 
negotiations.  The  organizational  interview  was  successfully  completed  with  88  of  the  98 
establishments  (90%)  (8  more  refusals  occurred  at  this  stage  and  2 companies  went  out-of- 
business).  At  the  employer  stage,  a total  of  34  establishments  (39%)  returned  at  least  one 
completed  diskette  and  54  (61%)  neither  refused  overtly  nor  mailed  back  completed  diskettes.  At 
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the  incumbent  stage,  the  response  rate  was  23%-a  total  of  247  completed  diskettes  were 
returned.  Figure  2-9  displays  the  response  rates  for  the  feasibility  study  at  all  stages  of  data 
collection. 

Identifying  Response  Rate  Problems  in  the  Feasibility  Study 

The  response  rates  for  the  feasibility  study  were  slightly  higher  than  those  obtained  in  the 
baseline  study.  However,  as  in  the  baseline  study,  several  suggestions  for  improving  the 
administration  process  were  provided  by  POCs  during  nonresponse  follow-up  telephone  calls. 
First,  while  the  survey  was  reduced  to  approximately  30  minutes  using  the  pilot  test  design,  it 
often  took  much  longer  than  that  to  complete  the  same  questionnaire  using  the  computer.  The 
longer  time  for  completion  may  in  part  be  attributable  to  the  loading  and  clean  up  time  necessary 
for  computer  administration. 

Employers’  reported  reasons  for  nonresponse  were  those  reported  in  the  baseline  study. 
Specifically,  many  of  the  employers  contacted  reported  that: 

• The  questionnaire  was  too  long; 

• They  had  already  provided  enough  information  by  participating  in  the  organizational 
interview; 

• Participation  in  the  study  required  too  much  administrative  effort  on  their  part;  and 

• Employee  participation  in  the  study  would  tie  up  group  computers  for  too  long  during  the 
work  day. 

Despite  disappointing  response  rates,  computer  administration  shows  promise  and  should 
be  considered  for  future  data  collection.  Employers  who  did  return  diskettes  often  reported  that 
incumbents  found  completing  the  questionnaire  to  be  less  tedious  and  more  interesting  than 
typical  paper-and-pencil  surveys. 
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Non-Response  Analysis 

Because  the  response  rates  in  this  study  were  low,  it  is  possible  that  the  survey  results 
obtained  are  seriously  biased.  That  is,  the  responding  employees  might  include  or  exclude 
particular  kinds  of  people  or  people  employed  in  particular  industries.  If  that  were  the  case, 
appropriate  caveats  would  need  to  be  stated  prior  to  any  discussion  of  findings.  In  order  to 
partially  address  the  issue  of  bias,  a nonresponse  analysis  was  conducted. 

Only  selective  evidence  has  been  uncovered  indicative  of  systematic  survey  response  or 
nonresponse.  The  size  of  establishments,  as  measured  by  number  of  employees  and  job  titles,  is 
related  to  survey  nonresponse  as  are  some  occupational  codes.  Additionally,  a firm’s  final  survey 
administration  disposition  is  correlated  with  one  of  the  questionnaire’s  series  of  questions-those 
measuring  high  performance  human  resources  practices.  Given  the  number  of  comparisons  made 
in  this  analysis,  however,  these  departures  are  considered  instructive  rather  than  pervasive. 

The  analyses  presented  here  cannot  directly  address  the  issue  of  bias.  The  most  these 
analyses  can  produce  is  indications  of  systematic  nonresponse  patterns.  Some  were  uncovered. 
However,  these  indications  are  limited  in  number  and  scope.  The  findings  from  this  analysis  are 
summarized  below. 

Non-Response  Analysis  Procedures 

A total  of  1,240  cases  were  provided  for  the  nonresponse  analysis.  Figure  2-10 
summarizes  the  distribution  of  cases  by  their  final  survey  disposition.  The  preponderance  of 
nonrespondents  and  refusals  underscores  the  need  for  examining  establishment  characteristics  by 
disposition  to  determine  whether  systematic  differences  exist  by  category.  For  this  analysis,  cases 
found  to  be  ineligible  during  any  phase  of  the  data  collection  effort  were  dropped,  leaving  a total 
of  1 ,054  cases  to  be  analyzed. 
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The  analysis  proceeded  in  two  stages.  First,  establishment  characteristics  known  for  all 
cases  were  examined.  These  included  the  SIC  code  of  the  establishments,  the  number  of 
employees  at  the  selected  location,  a status  indicator  (branch  or  HQ),  and  the  state  in  which  the 
establishment  was  located.  Each  of  these  items  was  supplied  by  D&B.  The  distribution  of  these 
characteristics  across  disposition  categories  was  examined  to  determine  whether  systematic 
concentrations  of  particular  firm  types  were  occurring  in  particular  disposition  categories. 

The  second  stage  of  the  analysis  examined  survey  results  also  arrayed  by  disposition 
status.  Figure  2-1 1 displays  the  distribution  of  completed  surveys  by  disposition  status.  This  table 
shows  that  the  majority  of  establishments  did  not  respond  at  the  mailout  phase  of  the  project. 

Several  variations  of  the  analyses  reported  here  were  performed.  For  example,  in  the 
analysis  of  firm  characteristics  by  disposition,  one  categorization  of  firm*;  collapsed  all  firms 
associated  with  a completed  questionnaire  into  a single  category  - completed  questionnaire.  The 
distribution  of  this  category  on  establishment  characteristics  was  then  compared  to  the  total 
sample  and  other  dispositions.  In  addition,  many  data  transformations  and  exploratory  linear 
regressions  and  analyses  of  variance  were  also  conducted  in  an  attempt  to  uncover  systematic 
patterns  of  nonresponse.  None  of  these  complementaiy  analyses  produced  findings  essentially 
different  from  those  that  are  presented  here  in  the  form  of  tables  and  bar  charts.  We  decided  that 
results  presented  plainly  were  more  valuable  than  those  employing  complex  transformations  or 
procedures  when  no  fundamental  alteration  in  results  was  obtained. 

Establishment  Characteristics  bv  Survey  Disposition 

D&B  provided  establishment  characteristics  for  firms  selected  in  the  sample.  Among  the 
characteristics  provided  were  the  geographic  location  of  the  firm  (state),  the  number  of 
employees  working  at  the  selected  location,  and  a status  indicator.  Figure  2-12  displays  the 
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distribution  of  firms  by  geographic  region  and  final  disposition  status.  In  this  figure  the 
percentage  expressed  is  the  percentage  occupying  a disposition  status.  For  example, 
approximately  24%  of  all  firms  refusing  at  negotiation  reside  in  the  North  Central  U.S.  while 
approximately  30%  reside  in  the  South.  The  sum  of  all  percentages  for  a disposition  equals 
100%. 

As  will  be  seen  in  many  of  the  figures  presented  in  this  nonresponse  analysis,  there  is 
general  parity  in  the  percentage  of  firms  in  each  disposition  category  within  region  and  greater 
variation  across  regions.  This  pattern  of  within  characteristic  similarity  and  across  region 
variation  is  the  primary  finding  of  the  nonresponse  analysis.  Within  geographic  region,  the 
percentages  of  firms  in  each  final  disposition  are  roughly  the  same,  while  greater  systematic 
variation  is  observed  between,  for  example.  Northeast  and  the  South. 

Figure  2-13  presents  final  disposition  by  firm  status  (e.g.,  whether  the  firm  exists  at  a 
single  location,  whether  the  location  selected  is  the  headquarters  or  a branch  office).  With  one 
major  exception,  the  distribution  of  firms  within  status  by  disposition  are  relatively  similar.  The 
exception  to  this  generalization  is  in  the  distribution  for  headquarters.  Here  there  are  far  greater 
numbers  of  mail  refusals  than  any  other  disposition  category.  While  between  20  and  25  percent 
of  all  other  dispositions  were  realized  at  firm  headquarters,  over  35  percent  of  all  mail  refusals 
occurred  at  firm  headquarters. 

The  distribution  of  dispositions  by  SIC  category  provides  some  distinctions  in  response 
and  nonresponse  patterns.  Figure  2-14  displays  final  disposition  by  SIC  categories.  These 
categories  were  formed  by  collapsing  similar  codes  into  larger  categories  such  as  construction, 
mining,  and  transportation.  The  justice  (justice,  public  order,  and  safety)  and  service  (health, 
educational,  and  social  services)  categories  distinguished  themselves  in  the  high  percentage  of 
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mail  completes  they  returned  compared  to  the  other  disposition  categories.  Conversely, 
construction  (building,  heavy  construction),  food  (food  and  kindred  products),  and  service 
(hotels,  personal,  and  business  services)  are  distinguished  by  their  relatively  small  percentage  of 
mail  completes  compared  with  the  other  dispositions  for  their  SIC  categories.  Survey  response 
varied  considerably  within  these  five  SIC  categories. 

Figure  2-15  presents  average  firm  size  (measured  as  number  of  employees)  by  disposition 
category.  For  the  total  sample,  the  average  firm  size  was  slightly  over  300  employees.  For  most  - ' 
disposition  categories  firm  size  fluctuated  about  the  sample  average.  The  average  size  of  firms 
providing  a “mail  refusal”  to  survey  administration  efforts  was  nearly  twice  the  sample  average. 

It  appears  that  mail  refusals  were  much  more  prevalent  for  larger  firms. 

Questionnaire  Factor  Scores  bv  Final  Survey  Disposition 

Following  the  examination  of  establishment  characteristics  by  final  disposition  category, 
attention  was  turned  to  questionnaire  responses.  As  shown  in  Figure  2-11,  nearly  one-half  of 
returned  questionnaires  were  associated  with  firms  having  a final  disposition  code  of  mail 
nonresponse.  The  purpose  of  this  portion  of  the  nonresponse  analysis  was  to  determine  whether 
systematic  differences  existed  in  survey  responses  by  final  disposition  category. 

A prominent  feature  of  the  CATI  organizational  survey  was  the  inclusion  of  questions 
designed  to  measure  concepts  related  to  organizational  structure,  human  resource  systems  and 
management,  and  organizational  values.  Operationally,  measurement  of  these  concepts  was 
accomplished  through  the  development  of  14  factor  scales.  Each  scale  was  associated  with 
multiple  survey  questions,  and  a scoring  algorithm  was  used  to  compute  factor  scale  scores. 

Figure  2-16  presents  average  factor  scores  for  each  of  the  scales  broken  out  by  final 
disposition  category.  Average  factor  scores  considered  significantly  different  than  those  for  the 
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other  dispositions  in  the  row  are  displayed  in  boldface  type.  Only  the  scales  measuring 
establishment  size  and  high  performance  exhibited  significant  variation  in  average  factor  scores. 
(As  the  distribution  of  some  factor  scales  are  highly  skewed,  their  distributions  were 
“normalized”  through  logarithmic  and  power  transformations.  As  the  results  of  comparisons  for 
the  transformed  scales  reproduced  those  obtained  in  the  original  metric,  we  limit  our  discussion 
to  the  original  metric.) 

The  establishment  size  factor  is  composed  of  Questions  concerning  the  respondent’s 
firm’s  size.  The  questions  used  to  construct  this  scale  asked  about  the  establishment’s  total 
number  of  employees,  number  of  job  titles,  and  number  of  specializations.  Respondents  from 
firms  categorized  as  mail  refusals  scored  between  70  and  80  percent  higher  on  this  factor  than 
respondents  in  the  other  categories.  Presumably,  these  are  respondents  from  larger  firms.  This 
finding  is  veiy  much  in  concert  with  the  earlier  finding  that  the  firms  in  the  category  “mail 
refusal”  were,  on  average,  nearly  twice  as  large  as  firms  in  the  other  final  disposition  categories. 

The  high  performance  human  resource  factor  is  composed  of  questions  regarding  the 
setting  of  goals.  This  constellation  of  questions  asks  about  managerial  and  non-managerial  goal 
setting  and  negotiation,  and  the  communication  of  these  goals.  For  this  factor,  the  “mail 
complete  category  differed  from  the  other  categories  in  that  its  average  factor  score  was 
approximately  80%  of  the  average  observed  for  “mail  nonresponse  and  refusal”  dispositions. 
There  does  not  appear  to  be  an  obvious  link  between  this  and  earlier  findings  concerning 
establishment  characteristics  and  final  survey  administration  disposition. 

The  other  12  factors  showed  relatively  small  differences  across  the  disposition  categories. 
Non-Response  Analysis  Summary 
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Although  analyses  such  as  these  summarized  here  cannot  definitively  establish  the 
presence  or  absence  of  nonresponse  bias  in  survey  findings,  they  can  provide  general  indications 
of  survey  response  coverage.  Generally,  it  was  established  that,  when  categorized  by  final 
disposition,  both  establishment  characteristics  and  questionnaire  responses  were  more  similar 
within  categories  than  across.  This  observation  is  tempered  by  findings  indicating  that 
establishment  size  and  occupational  grouping  do,  in  certain  cases,  impact  on  survey 
administration  outcome.  Larger  establishments  are  more  likely  than  others  to  terminate  the 
survey  process  through  “mail  refusal,”  and  certain  occupational  groupings  display  greater  or 
lesser  propensities  to  cooperate  and  provide  completed  surveys.  Regarding  high  performance 
practices  as  measured  via  a telephone  interview,  goal-setting  practices  showed  differences  across 
disposition  categories. 

Viewed  as  an  ensemble  of  findings,  the  nonresponse  analyses  provide  some  lesson 
learned.  Administration  of  an  establishment  survey  is  difficult  and  some  of  the  difficulty  is  a 
function  of  the  establishments  themselves.  Obviously,  establishment  size,  area  of  occupational 
focus  (e.g.,  construction,  service,  or  justice),  and  bureaucratic  structure  (as  reflected  in  human 
resource  practices)  affect  survey  response  differentially.  When  reviewing  survey  responses,  few 
indications  of  bias  were  observed.  Of  the  14  scales  evaluated  and  the  42  comparisons  made,  only 
2 scales  revealed  significant  differences  in  responses  by  disposition.  It  is  less  the  case  that  bias 
has  been  discovered  than  some  administrative  difficulties  with  the  procedures  have  been 
underscored.  Bearing  this  in  mind,  several  suggestions  can  be  made  for  improving  response  rate. 
Most  emphasized  is  the  suggestion  that  future  data  collection  should  not  be  limited  to  a single 
design  (e.g.,  stratified  random  sample).  Rather,  a mix  of  different  approaches  should  be 
considered  while  also  ensuring  that  the  outcomes  are  systematic  and  can  be  generalized  beyond  a 
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restricted  sample.  In  addition,  whatever  the  method  employed,  incentives  should  be  considered 
both  to  employers  and  to  employees.  Some  of  the  approaches  to  be  considered  include: 

• Within  the  random  sampling  approach,  more  specific  SIC  codes  should  be  used.  More 
specificity  (perhaps  at  the  6-digit  level)  should  help  in  identifying  establishments 
employing  people  in  less  populous  occupations  (e.g.,  instrumental  musician, 
photographer)  who  work  in  establishment  settings.  However,  other  methods  should  be 
used  to  locate  individuals  whose  occupations  do  not  typically  cluster  in  establishments.  A 
sample  could  be  drawn  from  lists  such  as  those  available  through  professional 
associations. 

• Reducing  establishment  burden  should  be  considered,  pending  examination  of  more 
occupations  using  this  traditional  sampling  design  with  the  reduced  burden  features  (e.g., 
reduced  questionnaires,  fewer  employees  required  within  a company,  no  formal 
sampling). 

► Another  short-term  action  involves  collecting  data  from  establishments  who  have  already 
expressed  a willingness  to  support  the  development  of  the  0*NET  database.  Several 
establishments  have  contacted  Department  of  Labor  personnel  and  expressed  an  interest 
in  participating  in  the  study  and  using  the  results  obtained  from  0*NET.  POCs  could  be 
identified  at  these  establishments  and  participation  solicited.  Occupations  within  these 
“volunteer”  establishments  would  then  be  determined,  and  the  POCs  would  distribute 
employee  questionnaires  following  the  procedures  previously  described.  However,  a 
procedure  for  disseminating  establishment-specific  results  or  using  the  0*NET  database 
would  need  further  development. 
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• Opportunistic  data  would  be  collected  by  soliciting  the  cooperation  of  volunteer 

organizations.  This  would  enable  the  formation  of  a consortium  of  large,  diversified 
employers  such  as  Bell  Atlantic,  General  Electric,  and  Marriott.  These  companies  employ 
people  across  a wide  range  of  jobs  and  have  the  most  to  gain  by  the  availability  of 
descriptive  information  about  jobs  in  these  respective  organizations.  In  addition,  a 
random  sampling  approach  might  be  used  to  collect  data  in  jobs  that  employ  large 
numbers  of  people. 

Data  Preparation 

As  a first  step  in  ensuring  data  quality,  all  returned  questionnaires  returned  were  cleaned 
and  coded  prior  to  data  entry.  Upon  receipt  of  incumbent  packets,  identification  numbers  were 
entered  into  a receipt  control  system  which  kept  track  of  which  questionnaires  were  completed  by 
each  incumbent.  As  the  mail  arrived,  it  was  opened  and  each  form  was  checked  to  make  sure  it 
had  at  least  some  responses.  A form  number  was  written  at  the  top  of  the  questionnaire  to  aid  in 
data  entry.  Refusals  were  also  entered  into  the  receipt  control  system.  If  a company  returned  all 
of  the  incumbent  packets  blank  or  indicated  a refusal  by  telephone,  this  information  was  entered 
into  the  database  by  company  name.  Following  this  step,  questionnaires  were  then  sorted  by  form 
type  and  coded. 

Codebook  Verification 

Codebooks  were  produced  for  each  data  collection  instrument.  The  codebook  assigns  a 
unique  variable  name  to  each  piece  of  data  collected,  identifies  the  column(s)  in  which  the  data 
are  stored,  and  defines  valid  responses  allowed  for  each  question.  Each  instrument  was  examined 
page  by  page  to  ensure  that  the  data  would  be  keypunched  correctly.  Responses  are  checked  for 
such  problems  as:  X’s  marked  in  any  place  other  than  over  a valid  response  number,  non-valid  or 
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out-of-range  responses,  respondents  marking  more  than  one  answer  when  only  one  should  be 
marked,  and  missed/improperly  skipped  items. 

Additionally,  some  of  the  questionnaires  required  coding  of  respondent  answers.  For 
example,  in  the  occupation-specific  questionnaires,  the  number  of  written  added  tasks  was 
entered  at  the  end  of  the  task  list.  If  no  additional  tasks  were  added  by  the  respondent,  a 0 was 
entered.  Similarly,  if  there  was  more  than  one  response  to  the  highest  level  of  education  on  the 
background  questionnaire,  the  highest  level  marked  was  the  response  coded.  For  example,  if 
Bachelor  s Degree  and  Master’s  Degree  were  both  marked,  the  Master’s  Degree  was  chosen  by 
the  coder  as  the  correct  response.  Finally,  if  an  “other  specified”  response  was  written  in  the 
sources  for  recruitment  for  current  job,  the  responses  were  typed  into  a separate  log. 

Data  Processing 

Once  the  coding  was  completed,  the  questionnaires  were  sent  to  data  entry  for 
keypunching.  All  questionnaires  were  key  punched  and  verified  by  experienced  data  entry 
operators.  Once  the  data  were  keyed,  various  edit  checks  were  run.  First,  the  data  were  checked 
to  ensure  responses  were  within  valid  ranges,  proper  skip  patterns  were  followed,  and  there  were 
no  missed  or  improperly  skipped  items.  When  errors  were  noted  by  the  edit  check  program,  each 
case  was  examined  and  corrections  were.made  by  updating  the  data  file  with  the  correct 
information. 

Following  these  edits,  data  quality  checks  were  made  and  rules  for  inclusion  of  data  were 
invoked.  These  checks  occurred  within  each  domain  or  questionnaire  (e.g.,  within  the  skills, 
within  generalized  work  activities).  If  a data  quality  check  detected  unusable  data  within  one 
domain,  it  had  no  effect  on  the  checks  in  another  domain. 
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If  a respondent  had  more  than  10  percent  missing  data  within  any  domain,  the  entire  case 
(i.e.,  all  of  the  respondent’s  responses  for  that  questionnaire)  was  deleted.  If  a respondent  had 
fewer  than  10  percent  missing  responses,  but  had  some  missing  responses,  then  the  mean 
descriptor  value  for  that  occupation  was  substituted  for  the  missing  response. 

An  occupation  was  retained  for  analyses  if  four  or  more  respondents  had  provided  data 
surviving  these  data  processing  and  quality  checks.  Although  our  goal  was  to  obtain  30 
respondents  for  each  domain  for  each  questionnaire,  four  respondents  was  sufficient  to  carry  out 
investigative  analyses  for  evaluative  purposes. 

When  these  rules  were  invoked,  there  were  29  occupations  that  had  at  least  four 
respondents  across  all  the  0*NET  content  domains.  These  occupations  are  listed  in  Chapter  13, 
which  describes  the  cross-domain  analyses.  There  were  somewhat  larger  numbers  of  occupations 
available  within  each  domain,  where  the  requirement  was  only  for  four  respondents  of  the 
relevant  questionnaire,  not  across  all  questionnaires;  those  occupations  are  displayed  in  the 
chapters  that  describe  their  respective  analyses. 

General  Analyses 

The  initial  0*NET  data  collection  effort  was  not  intended  to  provide  a comprehensive 
description  of  all  jobs  in  the  United  States  economy.  Instead,  it  was  part  of  a prototyping  effort 
intended  to  provide  evidence  bearing  on  the  meaningfulness  of  the  descriptive  information 
provided  by  these  measures.  Thus,  as  stated  in  the  beginning  of  this  chapter,  our  intent  in 
conducting  analyses  of  these  data  was  to  evaluate  the  initial  evidence  for  the  internal  and  external 
validity  of  the  resulting  descriptive  system. 

The  nature  of  the  measures  used  in  any  particular  domain  sometimes  dictated  unique 
kinds  of  analyses.  In  all  cases,  however,  the  basic  questions  of  concern  in  deciding  whether  these 


2-46  Peterson,  Mumford,  Levin,  Green,  & Waksberg 

measures,  in  fact,  appeared  to  yield  meaningful  inferences  dictated  a relatively  straightforward 
set  of  core  analyses  carried  out  within  each  domain 

The  within-domain  analyses  can  be  classified  into  seven  basic  types  of  analyses.  These 
are  generally  described  here;  details  specific  to  a domain  are  covered  in  their  respective  chapters. 

Descriptive  Statistics 

Means  and  standard  deviations  of  occupational  means  for  the  various  scales  within  each 
domain  are  provided  to  give  a general  impression  of  the  overall  centrality  and  variance  of  the 
descriptors  within  our  sample  of  occupations. 

Reliability 

By  reliability  we  primarily  mean  interrater  agreement.  No  rate-rerate  data  were  collected. 
We  computed  interrater  agreement  coefficients  for  each  descriptor  measure,  using  standard 
intraclass  coefficients  (Shrout  & Fleiss,  1979).  Such  coefficients  were  computed  for  the  level, 
importance,  frequency  or  other  scales  available  within  each  domain.  We  used  Spearman-Brown 
corrections  for  estimating  the  reliability  coefficients  for  the  1 -rater  and  30-rater  cases,  to  allow 
ready  comparisons  across  domains  and  to  other  studies.  We  used  the  harmonic  mean  of  the 
number  of  raters  available  for  each  occupation  as  our  estimate  of  k for  making  these  calculations. 
This  is  a conservative  estimate  of  k. 

Scoring 

In  some  cases— most  notably,  level  and  importance— different  methods  of  using  the 
responses  to  arrive  at  scale  scores  were  available.  These  methods  primarily  differed  in  the  way  in 
which  the  not  relevant  or  NR  response  was  used.  The  intended  and  default  method  was  to  use  not 
relevant  as  the  zero  point  on  the  level  scale  and  to  indicate  “not  important”  on  the  importance 
scale.  We  labeled  this  the  “full  scale”  method.  Alternatively,  we  examined  the  case  when  not 
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relevant  responses  were  treated  as  missing,  i.e.  only  those  selecting  not  relevant  were  not 
included  in  the  reliability  analyses.  Finally,  we  also  examined  a dichotomous  case  for  the  level 
scale,  where  not  relevant  = 0 and  any  other  response  was  coded  as  1 . These  analyses  were  done 
primarily  to  identify  any  differences  that  might  have  occurred  if  not  relevant  was  to  be  used  other 
than  as  it  was  intended. 

Analyses  of  Variance 

We  also  computed  an  analysis  of  variance  with  occupations  as  a between-raters  variable, 
and  descriptors  as  a within-raters  variable.  Aside  from  describing  the  sources  of  variance  in  the 
descriptor  measures,  these  analyses  allowed  a second,  stringent  method  of  computing  the 
interrater  agreement.  We  computed  the  intraclass  correlation  coefficient  using  the  descriptor  by 
occupation  source  of  variance  as  the  “true”  variance  and  the  descriptor  by  raters  within 
occupation  source  of  variance  as  the  “error”  source  of  variance.  Note  that  the  descriptor  source  of 
variance,  which  certainly  is  a non-error  source  of  variance,  is  excluded  from  the  calculation.  Only 
the  source  of  variance  that  serves  to  accurately  describe  each  occupation  as  it  differs  from  other 
occupations  is  included  as  true  variance.  This  is,  of  course,  the  most  appropriate  coefficient  for 
our  purposes,  but  it  is  most  likely  an  under-estimate  of  within-occupation,  interrater  agreement. 

This  kind  of  analysis  was  also  applied  to  higher-level  descriptor  scores  derived  by 
computing  the  mean  values  for  descriptors  combined  according  to  the  a priori  hierarchical 
structures  for  the  content  model  domains.  Since  these  higher-level  scores  would  no  doubt  be  used 
in  some  applications  of  the  0*NET  database,  it  was  useful  to  determine  the  interrater  agreement 


coefficients  for  such  scores. 
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All  of  these  reliability  analyses  are  by  way  of  evaluating  the  degree  to  which  incumbents 
appear  to  agree  on  the  description  of  their  occupations  using  the  0*NET  job  analysis  measures. 
Such  analyses  are  the  bedrock  upon  which  all  other  analyses  depend. 

Descriptor  and  Scale  Relationships 
Correlations  Between  Scale  Type 

In  most  domains,  more  than  one  scale  was  used  to  collect  information  about  descriptors. 
Most  notably,  level  and  importance  were  used  in  most  domains.  To  estimate  the  redundancy  of 
information  for  the  different  scales,  intercorrelations  were  computed  in  two  different  ways— 
across  descriptors  within  each  occupation  and  across  occupations  within  each  descriptor. 
Relationships  of  Descriptor  Scale  Scores 

To  assess  the  internal  structure  of  the  descriptor  variables,  correlations  were  computed 
between  the  level  scale  scores  and  between  the  importance  scale  scores,  or  other  appropriate 
scores.  These  correlations  were  computed  at  the  occupational  level  (i.e.,  occupational  means 
were  the  scores  that  were  correlated)  and  at  the  individual  level  (i.e.,  individual  incumbent  scores 
were  correlated,  but  only  four,  randomly  selected  incumbents  from  each  occupation  were  entered 
into  these  calculations  in  order  to  equally  weight  the  occupations).  Because  the  primary  focus  of 
the  0*NET  is  at  the  occupational  level,  only  the  occupational-level  data  were  examined  and 
discussed. 

Factor  Structure 

Principal  components  analyses  were  also  conducted  using  the  occupational-level 
correlational  data.  We  attempted  confirmatory  factor  analyses  in  some  cases,  but  those  attempts 
were  generally  not  useful  because  of  our  relatively  small  sample  sizes  (30-35  occupations  was 
the  N).  The  correlational  and  principal  component  analyses,  of  course,  provide  evidence  for 
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evaluating  the  meaningfulness  of  the  relationships  observed  among  the  measures  (Harvey, 
Friedman,  Hakel,  & Cornelius,  1982). 

Occupation  Differences  and  Discriminant  Analyses 
Two  kinds  of  analyses  were  conducted  here.  Six  representative  and  relatively  distinct 
occupations  were  selected  and  their  profiles  of  descriptor  scores  were  compared  within  each 
domain.  The  pattern  of  differences  across  these  occupations  was  used  to  illustrate  the  ways  in 
which  each  domain’s  descriptors  served  to  separate  occupations,  and  served  as  a check  on  the 
“sensibility”  of  the  resulting  occupational  descriptions.  The  second  analysis  used  discriminant 
function  analysis  with  the  descriptor  level  scores  (generally)  as  predictor  variables  and 
occupations  as  criterion  categories.  The  results  were  used  to  assess  how  well  these  measures 
could  discriminate  among  occupations,  and  the  nature  of  the  descriptors  yielding  the  best 
discrimination. 

Convergence  With  Analysts’  Ratings 

Because  we  had  incumbent  and  occupational  analyst  ratings  for  a subset  of  occupations, 
we  could  compare  these  ratings.  We  were  able  to  complete  these  analyses  for  five  of  the  nine 
content  domains,  since  analysts  had  completed  their  ratings  for  those  five  domains.  We 
calculated  and  displayed  the  means,  standard  deviations,  and  interrater  agreement  coefficients  for 
all  descriptors  for  both  the  level  and  importance  scales  in  these  five  domains,  except  for  the  work 
context  domain  which  used  a different  set  of  scales.  In  addition  we  calculated  t tests  for  mean 
differences,  F tests  for  variance  differences,  correlations  between  incumbent  and  analyst  mean 
descriptor  scores  (within  descriptor,  across  occupations),  and  mean  d2  values  (that  is,  the  mean 
across  all  occupations  of  the  summed,  squared  differences  between  incumbent  and  occupational 
mean  values  for  all  descriptors  within  an  occupation).  This  set  of  statistics  allowed  us  to 
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thoroughly  evaluate  the  degree  to  which  the  two  types  of  raters  converged  or  diverged  in  the 
information  they  provided.  The  correlations  indicate  the  degree  of  agreement  in  the  pattern  of  the 
mean  ratings  while  the  d2  values  indicate  the  averaged,  absolute  level  of  disagreement  between 
mean  ratings  of  the  two  types  of  raters.  Thus,  large  correlation  coefficients  and  small  d2  values 
indicate  convergence  between  the  raters. 

We  also  conducted  principal  component  analyses  of  the  correlations  between  descriptor 
level  scores  for  the  analyst  data-that  is,  a parallel  analysis  of  that  completed  for  the  incumbent 
data.  These  two  solutions  were  compared  and  contrasted  to  shed  further  light  on  the  similarities 
and  differences  of  the  information  provided  by  the  two  types  of  raters. 

Additional  Validity  Evidence 

Other  analyses  were  conducted  within  some  of  the  domains.  These  were  aimed  at  further 
construct  validation  of  the  measures,  e.g.  the  skill  level  profiles  for  occupations  with  different 
levels  of  education  were  compared.  These  analyses  are  described  in  the  individual  chapters. 

Within-Domain  Analysis  Tables 

The  results  of  the  above-described  analyses  are  presented  in  the  domain  chapters-- 
Chapters  3 through  1 l~of  this  report.  Each  of  these  chapters  presents  the  results  of  analyses  for 
one  of  the  0*NET  questionnaires.  Figure  2-17  describes  the  tables  used  to  present  the  analysis 
results.  The  tables  within  these  chapters  are  named  and  numbered  uniformly,  for  ease  of  location 
and  comparison,  except  for  Table  17,  which  is  unique  to  each  chapter.  Not  all  tables  apply  to  all 
chapters,  and  tables  may  not  be  discussed  sequentially  within  a given  chapter. 

Other  Analyses 

Other  chapters  in  this  report  detail  important  sets  of  analyses  intended  to  further  evaluate 
the  meaningfulness  and  usefulness  of  the  0*NET  system.  Chapter  16  describes  the  initial 
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analyses  of  measures  of  occupation-specific  tasks  as  well  as  work  conducted  to  develop  an 
efficient  method  to  collect  knowledge,  skill,  and  tool  information  at  the  occupation-specific  level 
by  capitalizing  on  the  0*NET  content  model.  Chapter  15  describes  initial  investigations  at 
producing  job  descriptions  that  capitalize  on,  yet  are  not  overwhelmed  by,  the  richness  of  the 
0*NET  information  base.  Chapter  12  enumerates  and  discusses  possible  covariates  of  0*NET 
descriptor  ratings,  and  summarizes  the  available  data  with  respect  to  some  of  those  covariates. 
Chapter  13  describes  analyses  of  the  relationships  across  the  content  domains.  Chapter  14 
describes  analyses  conducted  to  evaluate  cluster  analysis  techniques  that  might  be  applied  to 
0*NET  data.  Each  of  these  chapters  thoroughly  describes  the  methods  used  and  results  obtained. 
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Figure  2-1 

Example  Pages  From  the  Skills  and  Generalized  Work  Activities  Questionnaires 
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Figure  2-2 

Example  Instructions  for  the  Skills  Questionnaire 


Instructions  for  Making  Skill  Ratings 

In  this  questionnaire  you  will  be  presented  with  a list  of  46  skills.  Some  of  the  skills  are  developed  over  time  and  are  used 
not  only  to  do  work  but  to  leam  other  skills;  examples  include  Reading  Comprehension,  Writing,  and  Speaking.  Other 
skills  are  important  for  performance  on  many  jobs;  examples  include  Idea  Generation,  Troubleshooting,  and  Time 
Management 

For  each  skill,  please  make  the  following  three  ratings:  LEVEL,  IMPORTANCE,  and  JOB  ENTRY  REQUIREMENT. 

(1)  LEVEL.  Ask  yourself.  What  level  of  this  skill  is  needed  to  perform  this  job?"  To  help  you  make  this  judgment,  the 
LEVEL  scale  includes  descriptions  of  activities  requiring  high,  medium,  and  low  levels  of  the  skill.  These  are  only 
examples,  so  they  may  or  may  not  apply  to  the  specific  job  you  are  describing. 

Use  the  example  descriptions  to  select  the  number  on  the  scale  that  indicates  the  skill  level  required  by  the  job,  and  mark 
through  the  appropriate  number,  from  1 (indicating  that  a very  low  level  of  the  skill  is  required)  to  7 (indicating  that  a very 
high  level  of  the  skill  is  required)  on  the  LEVEL  scale.  For  example,  the  level  of  "Reading  Comprehension"  needed  for 
one  job  might  be  much  higher  than  that  needed  for  another  job. 

THE  NOT-RELEV  ANT  (NR)  RESPONSE.  If  the  skill  is  NOT  RELEVANT  at  all  to  performance  on  the  job,  mark 
through  the  NR  circle  that  appears  at  the  bottom  of  the  LEVEL  scale.  Carefully  read  all  of  the  level  descriptions  before 
selecting  the  NR  option.  If  you  select  NR,  however,  there  is  no  need  to  complete  the  IMPORTANCE  and  JOB  ENTRY 
REQUIREMENT  ratings  described  below. 

(2)  IMPORTANCE.  (Complete  only  if  a 1 to  7 LEVEL  rating  was  selected.)  Ask  yourself,  "How  important  is  this  skill  to 
performance  on  this  job?  For  example, ' Information  Gathering"  might  be  very  important  for  one  job,  but  less  important 
for  another  job.  For  the  second  job,  however,  "Listening/Questioning"  might  be  very  important 

Rate  the  IMPORTANCE  of  the  skill  for  performance  on  the  job  by  marking  through  the  appropriate  number,  from  1 
(indicating  that  the  skill  is  of  little  or  no  importance)  to  5 (indicating  that  the  skill  is  very  important)  on  the 
IMPORTANCE  scale. 

(3)  JOB  ENTRY  REQUIREMENT.  (Complete  only  if  a 1 to  7 LEVEL  rating  was  selected.)  Bearing  in  mind  the 
LEVEL  of  the  skill  that  is  needed  to  perform  the  job,  ask  yourself,  "Is  this  level  of  the  skill  required  for  entry  to  this  job?" 
For  example,  "Reading  Comprehension"  might  be  needed  by  an  employee  before  starting  one  job.  However, 
’Troubleshooting"  might  not  be  necessary  before  starting  a different  job.  "Troubleshooting"  might  be  learned  during  on- 
the-job  training. . 

Rate  the  JOB  ENTRY  REQUIREMENT  for  the  job  by  marking  through  the  appropriate  number,  either  1 (indicating  that 
the  level  of  the  skill  is  required  for  entry  on  the  job)  or  2 (indicating  that  the  level  of  the  skill  can  be  learned  on  the  job).  ' 

Notice  that  the  LEVEL  of  a skill  and  the  IMPORTANCE  of  the  skill  are  different.  For  example,  "Reading 
Comprehension  can  be  high  in  IMPORTANCE  for  two  different  jobs,  but  the  LEVEL  of  "Reading  Comprehension"  that 
an  employee  in  one  job  needs  is  not  as  high  as  the  LEVEL  of  "Reading  Comprehension"  for  an  employee  in  another  job. 

[two  completed  examples  were  included  here] 


Turn  the  page  to  begin  the  Skills  Questionnaire. 


Figure  2-3 
Sampling  Stages 


Figure  2-4 

Counts  of  Occupations,  Total  Employment,  and  Sample  Size  bv  Job  Family 


CO 


CO 

1^. 


Figure  2-4  (continued) 

Counts  of  Occupations.  Total  Employment,  and  Sample  Size  by  Job  Family 


"I4B 

Protective 

Service  Occupations 

“140” 

Not  available 
Not  available 

“141” 

Not  available 
Not  available 

“142” 

9 

538,500 

0.26 

0.40 

1 

r*  3 ° ° 

VO 

“144” 

4 

895,070 

0.43 

0.67 

1 

2,095,530 

1.570051991 

3 

“12” 

Retail  and  Wholesale 
Sales  Occupations 

“120” 

1 

432,250 

0.03 

0.32 

1 

o 

— 2 m 

<N  rf  ~-  © ~ 

r o ° ° 

r- 

“122” 

10 

3,335,220 

0.26 

2.50 

2 

“123” 

4 

4,139,480 

0.32 

3.10 

2 

“124” 

5 

4,369,530 

0.34 

3.27 

3 

1 12,979,600 

9.724817502 
. 9 

“10” 

Art,  Culture  and 
Recreation  Occupations 

“100” 

Not  available 
Not  available 

“101” 

11 

491,270 

0.56 

0.37 

1 

“102” 

9 

356,120 

0.41 

0.27 

1 

r © 

co  - n <S 

o (s  t 9 9 o 

r“  mo© 

* 

“104” 

Not  available 
Not  available 

870,800 

0.652436984 

2 

“09” 

Education,  Training 
and  Instructional  Occ. 

“090” 

1 

298,680 

0.05 

0.22 

1 

“091” 

21 

4,403,740 

0.76 

3.30 

2 

“092” 

2 

706,950 

0.12 

0.53 

1 

“093” 

2 

365,080 

0.06 

0.27 

1 

5,774,450 

4.326440909 

5 

Distinguishing 

Activity 

Job  Family 
# Occupations 
Total  Employment 
Column  Percent 
E(n) 
n 

Job  Family 
# Occupations 
Total  Employment 
Column  Percent 
E(n) 
n 

Job  Family 
# Occupations 
Total  Employment 
Column  Percent 
E(n) 

n 

Job  Family 
# Occupations 
Total  Employment 
Column  Percent 
E(n) 
n 

Job  Family 
# Occupations 
Total  Employment 
Column  Percent 
E(n) 
n 

Column  Total 
E(n) 
n 

Preparation 

Level 

Management 

Education, 

Training 

or 

Experience 

University 

Degree 

Two  to  Three 
Years 

Post-Secondary 
Education  or 
Equivalent 

High 

School 

Degree 

Preferred 

High 
School 
Degree 
Not  Required 

t-4 
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Figure  2-4  (continued) 

Counts  of  Occupations,  Total  Employment,  and  Sample  Size  bv  Job  Family 


“19” 

Occupations  Unique 
to  Ag.,  Forest  and  Fishing 

. 

r O 

O ^ N « 

Os  ©i  O O o 

r"  r-  © © 

= o 

<N 

o\  rr  ^ o © o 

r*  odd 
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“193” 

12 

710,770 

0.75 

0.53 

1 

“194” 

3 

162,820 

0.17 

0.12 

0 

950,980 

0.712510936 

1 

“18” 

Transportation  and 
Equipment  Occupations 

“180” 

Not  available 
Not  available 

O 

cm  S2  cn  Os 

oo  <N  - O O o 

r s © © 
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„ oo 

rn  co  on 

oo  £ \o  crs  vo  m 

7*  On  o CM 
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CO 

o 
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3,856,280 

2.889273879 

3 

“17” 

Craft 

Occupations 

“'170” 

2 

193,050 

0.05 

0.14 

0 

“172” 

25 

2,403,950 

0.67 

1.80 

1 

“173” 

11 

282,450 

0.08 

0.21 

1 

“174” 

14 

698,080 

0.20 

0.52 

1 

3,577,530 

2.680423615 

3 

“15” 

Personal  and  Commercial 
Service  Occupations 

“150” 

2 

359,420 

0.03 

0.27 

1 

“151” 

Not  available 
Not  available 

“152” 

7 

1,337,660 

0.10 

1.00 

1 

“153” 

15 

4,230,730 

0.33 

3.17 

3 

“154” 

24 

6,965,810 

0.54 

5.22 

4 

12,893,620 

9.660397966 
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Distinguishing 

Activity 

Job  Family 
# Occupations 
Total  Employment 
Column  Percent 
E(n) 
n 

Job  Family 
# Occupations 
Total  Employment 
Column  Percent 
E(n) 
n 

Job  Family 
# Occupations 
Total  Employment 
Column  Percent 
E(n) 
n 

Job  Family 
# Occupations 
Total  Employment 
Column  Percent 
E(n) 
n 

Job  Family 
# Occupations 
Total  Employment 
Column  Percent 
E(n) 
n 

Column  Total 
E(n) 
n 

Preparation 

Level 

Management 

Education, 

Train**!* 

or 

Experience 

University 

Degree 

Two  to  Three 
Years 

Post-Secondary 
Education  or 
Equivalent 

High 

School 

Degree 

Preferred 

High 
School 
Degree 
Not  Required 
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Figure  2-4  (continued) 

Counts  of  Occupations,  Total  Employment,  and  Sample  Size  bv  Job  Family 


Totals 

19 

6,599,600 

8 

110 

13,816,740 

9 

269 

22,736,160 

16 

223 

35,681,200 

27 

60 

14,594,480 

10 

93,428,180 

70 

i 

£5 

"21” 

Occupations  Unique  to 
Manufacturing,  Process,  etc. 

"210” 

1 

211,670 

0.03 

0.16 

1 

"2 12" 
52 

2,551,710 

0.35 

1.91 

1 

o 
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r^  ^ocN 

• CO 

CO 
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7,323,340 

5.486929104 

5 

"20” 

Occupations  Unique  to 
Extractive  Operations 

"200” 

Not  available 
Not  available 

"202" 

97 

6,055,050 

0.95 

4.54 

4 

"203” 

8 

160,080 
0.03 
0.12 
. 0 

o 

Tl-  £ CO  <N 

p ~ o 

f»  Soo 

6,379,680 

4.779902595 

4 
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Distinguishing 

Activity 

Job  Family 
# Occupations 
Total  Employment 
Column  Percent 
E(n) 
n 

Job  Family 
# Occupations 
Total  Employment 
Column  Percent 
E(n) 
n 

Job  Family 
# Occupations 
Total  Employment 
Column  Percent 
E(n) 
n 

Job  Family 
# Occupations 
Total  Employment 
Column  Percent 
E(n) 
n 

Job  Family 
# Occupations 
Total  Employment 
Column  Percent 
E(n) 
n 

Column  Total 
E(n) 
n. 

Preparation 

Level 

Management 

Education, 

Training 

or 

Experience 

University 

Degree 

Two  to  Three 
Years 

Post-Secondary 
Education  or 
Equivalent 

High 

School 

Degree 

Preferred 

High 
School 
Degree 
Not  Required 

O 

ERIC 


Figure  2-5 

Eighty  Occupations  Sampled  in  Initial  Q*NET  Data  Collection 


Job 

Family 

Occupation 

Code 

Occupation  Title 

Total 

Employment 

020 

19005 

General  Managers  & Top  Executives 

2,868,700 

020 

13002 

Financial  Managers 

716,050 

020 

13014 

Administrative  Managers 

248,210 

021 

21108 

Loan  Officers  & Counselors 

200,060 

♦021 

25315 

Financial  Analysts,  Statistical 

29,960 

022 

51002 

First  Line  Supervisors,  Clerical  & Administrative 

1,229,930 

023 

55347 

General  Office  Clerks 

-2,660,890 

023 

55108 

Secretaries,  except  Legal  & Medical 

2,440,560 

023 

55338 

Bookkeeping,  Accounting,  & Auditing  Clerks 

1,812,510 

023  . 

55305 

Receptionists  & Information  Clerks 

927,730 

023 

55307 

Typists,  including  Word  Processing 

612,490 

023 

53102 

Tellers 

554,640 

023 

55344 

Billing,  Cost  & Rate  Clerks 

319,940 

023 

56011 

Computer  Operators,  except  Peripheral  Equipment 

226,240 

023 

53121 

Loan  & Credit  Clerks 

178,860 

023 

53311 

Insurance  Claims  Clerks 

104,190 

023 

57105 

Directory  Assistance  Operators 

27,270 

050 

13017 

Engineering,  Mathematical,  & Natural  Sciences  Manager 

348,760 

051 

22302 

Architects,  Except  Landscape  & Marine 

60,070 

*051 

22114 

Chemical  Engineers 

53,930 

*051 

22127 

Computer  Engineers 

207,490 

*051 

22135 

Mechanical  Engineers 

228,850 

*051 

25102 

Systems  Analysts 

39,354 

*051 

25105 

Computer  Programmers 

448190 

052 

32905 

Medical  & Clinical  Laboratory  Technicians 

100,600 

*052 

32902 

Medical  & Clinical  Laboratory  Technologists 

148,800 

061 

32502 

Registered  Nurses 

1,764,950 

061 

31114 

Nursing  Instructors 

46,430 

062 

32926 

Electrocardiograph  Technicians 

15,870 

063 

66008 

Nursing  Aides,  Orderlies,  & Attendants 

1,1 17,980 

063 

66005 

Medical  Assistants 

198,090 

071 

31502 

Librarians,  Professional 

130,180 

072 

27311 

Recreation  Workers 

191,050 

090 

15005 

Education  Administrators 

298,680 

091 

31305 

Teachers,  Elementary  School 

1,313,510 

091 

31303 

Teachers,  Preschool 

356,740 

092 

31321 

Instructors  & Coaches,  Sports  & Physical  Training 

257,680 

093 

53905 

Teachers’  Aides  & Assistants,  Clerical 

355,580 

101 

34051 

Musicians,  Instrumental 

47,140 

102 

34023 

Photographers 

53,910 

120 

13011 

Marketing/Advertising/Public  Relations  Managers 

432,250 

121 

49002 

Sales  Engineers 

66,000 

♦Occupations  in  the  High  Performance  sample 
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Figure  2-5  (continued) 

Eighty  Occupations  Sampled  in  Initial  Q*NHT  Data  Collection 


Job 

Family 

Occupation 

Code 

Occupation  Title 

Total 

Employment 

122 

49008 

Salespersons,  Except  Scientific  & Retail 

1,167,870 

122 

21302 

Buyers,  Except  Farm  Products 

157,380 

123 

49011 

Salespersons,  Retail 

3,438,510 

123 

49014 

Salespersons,  Parts 

287,910 

124 

49023 

Cashiers 

2,660,370 

124 

49021 

Stock  Clerks,  Sales  Floor 

1,088,520 

124 

49017 

Counter  & Rental  Clerks 

323,340 

142 

• 61005 

Police  & Detective  Supervisors 

84,860 

143 

63014 

Police  Patrol  Officers 

380,230 

144 

63047 

Guards  & Watch  Guards 

829,530 

150 

15026 

Food  Service/Lodging  Managers 

337,120 

152 

65026 

Cooks,  Restaurant 

573,510 

153 

65008 

Waiters  & Waitresses 

1,748,910 

153 

65005 

Bartenders 

369,830 

153 

53805 

Reservation  & Transportation  Ticket  Agents 

116,180 

154 

67005 

Janitors  & Cleaners 

1,806,380 

154 

65038 

Food  Preparation  Workers 

1,194,610 

154 

65041 

Combined  Food  Preparation  & Service  Workers 

1,116,790 

154 

68014 

Amusement  & Recreation  Attendants 

230,650 

172 

87814 

Structural  Metal  Workers 

41,780 

173 

87902 

Earth  Drillers,  Except  Oil  & Gas 

12,170 

174 

98312 

Helpers,  Carpenters 

155,540 

183 

97102 

Truck  Drivers,  Heavy  or  Tractor-Trailer 

1,259,450 

183 

97105 

Truck  Drivers-Light,  Include  Delivery/Route  Workers 

884,040 

183 

97111 

Bus  Drivers,  Schools 

381,540 

193 

79855 

General  Farmworkers 

175,290 

202 

85132 

Maintenance  Repairers,  General  Utility 

1,118,560 

202 

85302 

Automotive  Mechanics 

563,960 

202 

85119 

Other  Machinery  Maintenance  Mechanics 

50,230 

202 

85123 

Millwrights 

68,720 

*202 

85705 

Data  Processing  Equipment  Repairers 

79,520 

*202 

89108 

Machinists 

343,780 

210 

15014 

Industrial  Production  Managers 

211,670 

212 

83002 

Precision  Inspectors,  Testers,  & Graders 

180,220 

213 

92974 

Packaging  & Filling  Machine  Operators 

324,910 

213 

89802 

Slaughterers  & Butchers 

60,020 

*213 

93905 

Electrical  & Electronic  Assemblers 

213,410 

214 

93938 

Meat,  Poultry  & Fish  Cutters  & Trimmers,  Hand 

127,820 

*Occupations  in  the  High  Performance  sample 


Note:  The  occupation  code  for  Teachers,  Preschool  was  corrected  from  31302  to  31303. 


Figure  2-6 
Rotation  Design 
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Figure  2-7 

Phase  I.  Baseline  (Paper  and  Pencil-)  Response  Counts 


SCREENING  (Receptionist) 

Total  to  Screening 
Ineligible1' 

Possible  Ineligible2* 

Refusal 

Complete 

1,240 

143 

42 

1 

1,054 

Response  Rate 

100% 

Survival  Rate 

MMI 

NEGOTIATIONS  (Human  Resources  Representative) 

Total  to  Negotiation 
Ineligible1' 

1,054 

80 

Refusal 

218 

Complete 

756 

Response  Rate 

ORGANIZATIONAL  REPRESENTATIVE  INTERVIEW  (Human  Resources  Representative) 

Total  to  Interview 

756 

Ineligible1* 

3 

Refusal 

92 

Complete 

661 

Response  Rate 

MAILOUT:  EMPLOYER  LEVEL  (mailed  18  October  to  15  December) 

Total  to  Mailout 

661 

Ineligible1* 

2 

Refusal 

174 

Nonresponse 

304 

Complete 

181 

Response  Rate 

MAILOUT:  TOTAL  EMPLOYEES 

Total  to  Mailout 

15,529 

Ineligible1' 

25 

Refusal/Nonresponse 

13,015 

Complete 

2,489 

Response  Rate 

MAILOUT:  EMPLOYEES  (employers  who  have  returned  data) 

Total  Employers  Who  Have  Returned  Data 

181 

Total  Employee  Packets  Sent 

4,125 

Total  Employee  Packets  Returned 
Response  Rate 

2,489 

MAILOUT:  EMPLOYEES  (employers  who  have  returned  data) 

Total  Employers  Who  Have  Returned  Data  with  at  least  1 General  Manager  or  Secretary 

96 

Total  Employee  Packets  Sent  - General  Managers 

198 

Total  Employee  Packets  Returned  - General  Managers 

145 

Response  Rate  - General  Managers 

■90 


Figure  2-8 

Phase  II  Pilot  Test  (Reduced  Burden  Paper  and  Pencil')  Response  Counts 


SCREENING(Receptionist) 

Total  to  Screening 

175 

Ineligible1* 

4 

Possible  Ineligible2* 

2 

Refusal 

0 

Complete 

169 

Response  Rate 

100% 

Survival  Rate 

mmmmm 

NEGOTIATIONS(Human  Resources  Representative) 

Total  to  Negotiation 

169 

Ineligible1* 

23 

Refusal 

11 

Complete 

135 

Response  Rate 

92% 

MAILOUT:  EMPLOYER  LEVEL(maiied  1 December  to  15  January) 

Total  to  Mailout 

135 

Ineligible1* 

8 

Refusal 

10 

Nonresponse 

52 

Complete 

65 

Response  Rate 

wsKsasm 

MAILOUT:  TOTAL  EMPLOYEES 

Total  to  Mailout 

457 

Ineligible1* 

0 

Refusal/Non  response 

271 

Complete 

186 

Response  Rate 

MAILOUT:  EMPLOYEES  (employers  who  have  returned  data) 

Total  Employers  Who  Have  Returned  Data 

65 

Total  Employee  Packets  Sent 

219 

Total  Employee  Packets  Returned 

186 

Response  Rate 

MAILOUT:  EMPLOYEES  (employers  who  have  returned  data) 

. Total  Employers  Who  Have  Returned  Data 

65 

Total  Employee  Packets  Sent  - General  Managers 

92 

Total  Employee  Packets  Returned  - General  Manaqers 

79 

Response  Rate  - General  Managers 

86% 

Total  Employee  Packets  Sent  - Secretaries 

127 

Total  Employee  Packets  Returned  - Secretaries 

107 

Response  Rate  - Secretaries 

i.  . 

Ineligible  cases  include:  duplicates;  fewer  than  5 employees;  English  not  spoken;  out  of  business;  no  one  in  chosen  occupations. 
a Possibly  ineligible  cases  include:  no  answer  at  location;  not  locatable;  phone  number  not  in  service. 


Figure  2-9 

Phase  I.  Feasibility  (DOS-Based  Software)  Response  Counts 


Total  to  Screening 
Ineligible1' 

Possible  Ineligible2* 

Refusal 

Complete 

SCREENING(Receptionist) 

200 

18 

5 

0 

177 

Response  Rate  , 

100% 

Survival  Rate 

HM 

NEGOTIATlONS(Human  Resources  Representative) 

Total  to  Negotiation 

177 

Ineligible1* 

54 

Refusal 

25 

Complete 

98 

Response  Rate 

ORGANIZATIONAL  REPRESENTATIVE  INTERVIEW(Human  Resources  Representative) 

Total  to  Interview 

98 

Ineligible1* 

2 

Refusal 

8 

Complete 

88 

Response  Rate 

SISiSI92^ 

MAILOUT:  EMPLOYER  LEVEL  (mailed  18  October  to  15  January) 

Total  to  Mailout 

88 

Ineligible1* 

0 

Refusal 

0 

Nonresponse 

54 

Complete 

34 

Response  Rate 

MAILOUT:  TOTAL  EMPLOYEES 

Total  to  Mailout 

1,099 

Ineligible1' 

25 

Refusal/Nonresponse 

827 

Complete 

247 

Response  Rate 

1.  . -L_  f . . . 

Possibly  ineligible  cases  include:  no  answer  at  location;  not  locatable;  phone  number  not  in  service. 
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Figure  2-10 

Distribution  of  Cases  bv  Disposition  Category 


Disposition  Category 

Count 

Percent 

Negotiation  Refusal 

218 

20.7 

CATI  Refusal 

92 

8.7 

Mail  Nonrespondents 

323 

30.6 

Mail  Refusals 

174 

16.5 

Mail  Completes 

162 

15.4 

Ineligible  Any  Phase 

85 

8.1 

Figure  2- 11 

Distribution  of  Completed  Questionnaires  bv  Disposition  Category 


Disposition  Category 

Count 

Percent 

Mail  Nonrespondents 

304 

46 

Mail  Refusals 

174 

26 

Mail  Completes 

181 

27 

4 


Figure  2-12 

Final  Disposition  by  Geographical  Region 
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Figure  2-13 

Final  Disposition  by  Firm  Status 
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Figure  2-14 

Final  Disposition  by  Standard  Industrial  Classification 
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Figure  2-15 

Average  Number  of  Employees  bv  Disposition 
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Figure  2-16 

Average  Factor  Score  bv  Disposition  Category 


Factor 

Disposition  Category 

Mail 

Nonrespondents 

Mail 

Refusals 

Mail 

Completed 

Decentralization 

2.3 

2.3 

2.4 

Employee  Benefits 

62.4 

63.6 

63.4 

Establishment  Size 

110.0 

199.1 

118.7 

Extent  of  Training 

3.4 

3.1 

3.5 

Formalization 

3.6 

3.6 

3.7 

Goal  Setting 

40.4 

39.7 

40.0 

High  Performance 

25.9 

26.0 

21.2 

Information  Sharing 

51.4 

47.4 

48.9 

Organizational  Size 

4778.6 

3596.4 

3898.4 

People  Orientation 

5.5 

5.5 

5.6 

Precision/Quality 

5.4 

5.6 

5.3 

Risk  Taking  * 

5.0 

5.0 

5.0 

Stability 

5.3 

5.3 

5.2 

Use  of  Teams 

15.7 

15.3 

14.9 

Figure  2-17 

Tables  Appearing  in  Domain  Chapters 
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Chapter  3 

Basic  and  Cross-Functional  Skills: 

Evidence  for  the  Reliability  and  Validity  of  the  Measures 

Michael  D.  Mumford 
Norman  G.  Peterson 
Ruth  A.  Childs 

American  Institutes  for  Research 


Economics  and  changes  in  the  nature  of  work  have  led  to  a new  concern  with  work  force 
skills.  Policy  makers  ask  what  skills  must  be  developed  for  the  high  wage/high  skill  occupations 
of  the  future.  Educators  ask  what  skills  will  be  essential  in  the  emerging  information  age. 


Business  leaders  ask  how  skills  can  be  used  to  promote  the  design  of  more  flexible  person- 
oriented  job  structures. 

The  0*NET  occupational  information  system  is  designed  to  provide  the  basic  descriptive 
information  needed  to  address  these  and  a host  of  other  questions  about  worker  skills.  In  this 
chapter,  we  review  the  available  evidence,  beginning  with  the  reliability  and  validity  of  the 


measures  of  occupational  skill  requirements  used  in  the  initial  0*NET  data  collection  effort.  We 
also  examine  the  implications  of  these  initial  findings  for  refinement  and  extension  of  the 
0*NET  content  model.  Before  turning  to  the  relevant  data,  however,  we  first  review  the  0*NET 


JAJti 


3-2  Mumford,  Peterson,  & Childs 


skills  taxonomy  and  the  procedures  used  to  collect  information  about  occupational  skill 
requirements. 


The  0*NET  skill  measures  are  based  on  the  taxonomy  of  workplace  skills  proposed  by 
Mumford  and  Peterson  (1995).  Within  this  framework,  a skill  is  viewed  as  a set  of  procedures  for 
working  with  or  applying  knowledge  within  a central  performance  domain.  The  0*NET  skills 
are  intended  to  capture  those  developed  performance  capabilities  that  can  be  viewed  as  cross- 
occupation  descriptions  in  the  sense  that  these  skills  are,  at  least  to  some  extent,  relevant  to 
describing  performance  in  many  different  occupations. 

Taking  these  propositions  as  starting  points,  Mumford  and  Peterson  (1995)  defined  skills 
with  respect  to  general  performance  domains  likely  to  be  involved  in  virtually  all  occupations. 
Development  of  their  taxonomy  began  with  the  notion  that  the  procedures  involved  in  acquiring 
and  conveying  information  are  part  of  all  occupations.  Thus,  basic  skills  such  as  Reading 
Comprehension  (Descriptor  #1),  Active  Listening  (Descriptor  #2),  Writing  (Descriptor  #3), 
Speaking  (Descriptor  #4),  Mathematics  (Descriptor  #5),  and  Science  (Descriptor  #6)  are 
expressly  included  in  the  0*NET  content  model.  In  addition,  Mumford  and  Peterson  (1995) 
argue  for  the  importance  of  skills  likely  to  promote  ongoing  learning  across  peoples’  careers. 
These  learning  to  learn  or  learning  process  skills  include  Critical  Thinking  (Descriptor  #7), 

Active  Learning  (Descriptor  #8),  Learning  Strategies  (Descriptor  #9),  and  Monitoring  (Descriptor 

#10). 

It  is  not  enough  simply  to  acquire  and  convey  knowledge.  Workplace  performance 
requires  the  effective  application  of  knowledge.  Using  sociotechnical  systems  theory  as  a 
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framework  for  defining  the  major  domains  of  performance  likely  to  be  found  on  most  jobs, 
Mumford  and  Peterson  (1995)  identified  five  general  performance  domains  likely  to  occur  on 
most  jobs:  (1)  Social,  (2)  Problem  Solving,  (3)  Technical,  (4)  Systems,  and  (5)  Resource 
Management.  These  five  general  domains  provide  the  basis  for  identifying  the  cross-functional 
skills  included  in  the  0*NET  content  model. 

The  problem  solving  skills  were  intended  to  capture  key  procedures  involved  in  solving 
complex  reactional  problems.  The  problem  solving  skills  included  in  the  0*NET  taxonomy  were 
Problem  Identification  (Descriptor  #17),  Information  Gathering  (Descriptor  #18),  Information 
Organization  (Descriptor  #19),  Synthesis/Reorganization  (Descriptor  #20),  Idea  Generation 
(Descriptor  #21),  Idea  Evaluation  (Descriptor  #22),  Implementation  Planning  (Descriptor  #23), 
and  Solution  Appraisal  (Descriptor  #24).  Not  only  must  people  solve  problems  on  their  jobs,  they 
also  must  solve  problems  by  working  with  others  in  the  context  of  a broader  sociotechnical 
system.  The  social  skills  that  appeared  to  apply  across  a variety  of  occupations  included  Social 
Perceptiveness  (Descriptor  #11),  Coordination  (Descriptor  #12),  Persuasion  (Descriptor  #13), 
Negotiation  (Descriptor  #14),  Instructing  (Descriptor  #15),  and  Service  Orientation  (Descriptor 
#16).  The  systems  skills  included  Visioning  (Descriptor  #37),  Systems  Perception  (Descriptor 
#38),  Identification  of  Downstream  Consequences  (Descriptor  #39),  Identification  of  Key  Causes 
(Descriptor  #40),  Judgment  and  Decision  Making  (Descriptor  #41),  and  Systems  Evaluation 
(Descriptor  #42). 

In  addition  to  problem  solving  in  a dynamic  social  setting,  most  jobs  require  people  to 
work  with  technology  and  resources  to  provide  certain  products  or  services.  Mumford  and 
Peterson  (1995),  drawing  from  prior  studies  of  managerial  performance,  identified  four  resource 
management  skills,  including  Time  Management  (Descriptor  #43),  Management  of  Financial 
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Resources  (Descriptor  #44),  Management  of  Material  Resources  (Descriptor  #45),  and 
Management  of  Personnel  Resources  (Descriptor  #46).  In  the  case  of  technical  skills,  an 
examination  of  prior  empirical  studies,  along  with  the  development  of  a general  performance 
model,  led  to  the  identification  of  twelve  technical  skills,  including  Operations  Analysis 
(Descriptor  #25),  Technology  Design  (Descriptor  #26),  Equipment  Selection  (Descriptor  #27), 
Installation  (Descriptor  #28),  Programming  (Descriptor  #29),  Testing  (Descriptor  #30), 
Operation  Monitoring  (Descriptor  #31),  Operation  and  Control  (Descriptor  #32),  Product 
Inspection  (Descriptor  #33),  Equipment  Maintenance  (Descriptor  #34),  Troubleshooting 
(Descriptor  #35),  and  Repairing  (Descriptor  #36). 

Figure  3-1  describes  the  46  basic  and  cross-functional  skills  included  in  the  0*NET 
content  model.  Mumford  and  Peterson  (1995)  provided  some  compelling  support  for  this  model 
of  the  skills  domain.  For  example,  they  showed  that  skills  included  in  each  domain  could  be 
organized  into  performance  models  which  found  strong  support  in  the  broader  theoretical 
literature.  Further  evidence  for  the  meaningfulness  of  this  taxonomy  was  provided  by  showing 
that  the  skills  included  in  this  taxonomy  accounted  for  more  than  90%  of  the  skills  variables 
identified  in  earlier  taxonomic  efforts. 

However  impressive  this  evidence  for  the  substantive  validity  of  the  0*NET  basic  and 
cross-functional  skills,  the  initial  Mumford  and  Peterson  (1995)  study  provided  little  empirical 
evidence  for  the  meaningfulness  of  this  taxonomy.  Moreover,  although  they  propose(d)  measures 
for  assessing  occupational  skill  requirements  with  respect  to  the  variables  included  in  this  model, 
evidence  bearing  on  the  reliability  of  these  measures  was  not  provided.  In  the  following  section 
we  briefly  review  the  procedures  used  to  measure  occupational  skill  requirements,  before  going 
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on  to  consider  the  reliability  and  validity  of  the  descriptive  data  resulting  from  use  of  these 
measures. 


A detailed  description  of  the  procedures  used  to  measure  skill  requirements  within  the 
0*NET  system  has  been  provided  by  Mumford  (1995)  and  Mumford  and  Peterson  (1995). 
Broadly  speaking,  the  approach  used  to  measure  occupational  skill  requirements  is  based  on 
earlier  work  (e.g.,  Fleishman,  1982;  Fleishman  & Quaintance,  1984)  concerned  with 
identification  of  the  abilities  held  to  determine  job  performance. 

Initially,  incumbents  or  analysts  are  presented  with  a definition  of  the  skill  written  in 
operational  terms.  After  reviewing  this  definition  they  are  asked  to  rate  the  level  at  which  this 
skill  is  required  for  a job,  using  a seven-point  behaviorally-anchored  rating  scale.  These  level 
ratings  are  intended  to  reflect  the  amount  of  the  skill  someone  must  possess  to  perform  requisite 
job  assignments.  In  making  these  level  ratings,  incumbents  or  analysts  are  asked  to  review  the 
anchors  reflecting  different  levels  of  skill  requirements  and  then  choose  the  rating  that  best 
corresponds  to  the  activities  that  occur  on  the  target  occupation.  Figure  3-2  provides  an 
illustration  of  these  rating  scales. 

The  anchors  used  to  obtain  these  level  ratings  were  developed  based  on  an  a priori 
analysis  of  the  nature  of  a skill  and  its  implications  for  performance  in  different  settings  (Childs 
& Whetzel,  1995;  Mumford  and  Peterson,  1995).  The  meaningfulness  of  these  anchors  was  then 
assessed  in  a categorization  task,  where  analysts  assigned  anchors  to  skilly  and  in  a rating  task, 
where  analysts  rated  the  level  of  the  skill  called  for  by  an  anchor.  Although  the  results  obtained 
in  these  scaling  studies  generally  provided  support  for  the  anchors,  they  indicated  the  need  to 
modify  or  change  a few  anchors—typically  low  end  anchors.  These  changes  were  made  by  having 
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analysts  generate  remedial  anchors  and  then  checking  those  anchors  in  a second  set  of  scaling 
studies. 

After  rating  the  level  of  skill  requirements,  two  other  ratings  were  made  if  it  was 
indicated  that  the  skill  was  relevant  to  understanding  performance  on  the  occupation  at  hand. 
First,  the  importance  of  the  skill  to  performance  on  the  job  was  rated  using  a five-point  scale.  The 
hypothesis  here  was  that  sometimes  a high  level  of  a skill  might  be  required  but  the  skill  might 
not  be  crucial  to  job  performance  (e.g.,  reading  for  stock  brokers).  Second,  after  making 
importance  ratings,  the  judges  were  asked  to  indicate  whether  this  level  of  skill  was  required  at 
job  entry  or  if  it  could  be  acquired  later  as  a function  of  job  experience.  This  dichotomous  rating 
was  intended  to  indicate  the  degree  to  which  skill  development  could  occur  on  the  job. 

In  our  initial  study,  two  sets  of  judges  were  used  to  describe  various  occupations  using 
these  rating  scales.  The  first  sample  of  judges  consisted  of  incumbents— people  working  on  the 

t 

job-who  were  asked  to  describe  the  requirements  of  their  job  using  these  rating  scales.  This 
incumbent  sample  was  obtained  through  a stratified  random  sampling  of  80  occupations  (out  of 
77 1 occupations)  representing  46  percent  of  the  employed  U.S.  population  (see  Chapter  2 for  a 
complete  description  of  the  sampling  procedures).  The  initial  data  collection,  using  a stratified 
random  sampling  of  establishments  with  the  targeted  occupations,  yielded  35  occupations  with  at 
least  four  incumbents  completing  the  relevant  skill  measures.  Figure  3-3  lists  these  occupations 
and  the  number  of  incumbents  who  completed  the  skill  ratings.  The  following  analyses  are  based 
on  the  responses  of  648  incumbents  from  138  establishments  or  organizational  sites.  It  should  be 
noted  that  the  counts  of  incumbent  respondents  included  some  supervisors;  however,  for 
simplicity,  they  will  be  referred  to  simply  as  incumbents. 
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In  addition  to  job  incumbents,  occupational  analysts  were  also  asked  to  describe  the 
characteristics  of  occupations  using  these  skill  scales.  The  analysts’  ratings  were  obtained  with 
the  cooperation  of  the  five  Occupational  Analysis  Field  Centers  (OAFCs).  Here,  a sample  of  at 
least  six  analysts  was  asked  to  review  a list  of  the  seven  to  30  most  important  tasks  found  in  an 
occupation  as  indicated  by  the  job  descriptions  provided  in  the  Dictionary  of  Occupational  Titles 
(DOT;  U.S.  Department  of  Labor,  1991).  Ratings  of  skill  requirements  were  obtained  for  all  80 
occupations  included  in  the  incumbent  sample.  To  permit  incumbent-analyst  comparisons,  only 
the  35  occupations  for  which  incumbent  data  were  also  available  were  considered  during 
analyses  of  the  analyst  data. 

Results 

Descriptive  Statistics 

Table  3-1  presents  the  basic  descriptive  statistics  obtained  for  each  skill  scale  when 
ratings  of  occupational  skill  requirements  were  obtained  in  the  incumbent  sample.  This  table 
presents  the  overall,  cross-occupation  means  and  standard  deviations  of  the  ratings  where 
occupations  were  treated  as  the  unit  of  analysis.  The  associated  interrater  agreement  coefficients 
and  standard  errors  of  measurement  are  also  presented. 

Turning  first  to  the  basic  descriptive  statistics,  one  clear  trend  emerged  which  applied  for 
all  rating  scales.  Typically,  basic  skills,  problem  solving  skills,  and  social  skills  had  higher  mean 
ratings  on  the  level  and  importance  scales  than  did  technical  skills,  systems  skills,  and  resource 
management  skills.  Additionally,  the  job  entry  requirement  ratings  produced  a similar  pattern  of 
findings  in  the  sense  that  they  indicated  that  the  requisite  level  of  basic,  problem  solving,  and 
social  skills  must  be  present  at  the  time  of  job  entry,  while  technical,  systems,  and  resource 
management  skills  could  be  acquired  later  as  people  worked  on  the  jobs. 


3-8  Mumford,  Peterson,  & Childs 


With  regard  to  variation  in  the  ratings  of  skill  requirements,  the  pattern  of  findings 
obtained  in  the  incumbent  analyses  was  typical  of  these  types  of  scales.  On  the  level  scales, 
ratings  vaned  from  one  to  one  and  a half  scale  points  on  a seven  point  scale.  This  variation  is 
typical  of  that  found  for  well-developed  level  scales  (Fleishman  & Mumford,  1991).  Importance 
ratings  were  obtained  on  a five-point  scale  and  standard  deviations  lay  between  a half  and  a full 
scale  point.  On  the  job  entry  requirement  scale,  a dichotomous  scale,  standard  deviations  were,  of 
course,  smaller,  ranging  between  .15  and  .25. 

Reliability 

Although  variation  was  observed  between  occupations  in  the  incumbents’  skill  ratings, 
the  question  arose  as  to  whether  this  variation  reflected  meaningful  differences  across 
occupations.  Some  initial  evidence  bearing  on  the  ability  of  these  scales  to  capture  meaningful 
variation  is  provided  by  the  standard  errors  of  measurement,  and,  more  centrally,  the  interrater 
agreement  coefficients  obtained  forjudges  drawn  from  the  same  job.  These  interrater  agreement 
coefficients  are  based  on  a harmonic  mean  of  9.01  judges  per  occupation  and  are  presented  in 
Table  3-1,  along  with  the  associated  standard  error  of  measurement. 

As  may  be  seen,  the  interrater  agreement  coefficients  for  the  level  scales  ranged  from  .75 
to  .92,  with  most  coefficients  lying  in  the  .80s.  These  are  excellent  interrater  agreement 
coefficients,  particularly  given  the  relatively  small  sample  of  judges  available.  The  interrater 
agreement  coefficients  obtained  for  the  importance  scale  produced  a similar  pattern  of  results. 
Again,  agreement  coefficients  lay  in  the  .80s,  ranging  from  .71  to  .93.  It  is  of  note  that  these 
results  did  not  appear  to  vary  noticeably  as  a function  of  scale  type.  Thus,  similar  coefficients 
were  obtained  for  the  basic  and  cross-functional  skills. 
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The  interrater  agreement  coefficients  obtained  for  the  job  entry  requirements  scale  were 
somewhat  lower.  Here,  agreement  coefficients,  across  scales,  lay  in  the  .70s,  ranging  between  .61 
and  .86.  Some  caution  should  be  exercised,  however,  when  contrasting  these  agreement 
coefficients  to  those  obtained  for  the  level  and  importance  scales,  since  the  smaller  agreement 
coefficients  obtained  for  the  job  entry  requirements  scale  may  simply  reflect  an  artifact  of  the  use 
of  a dichotomous  scoring  procedure. 

Overall,  these  initial  analyses  indicate  that  the  skill  scales  yielded  adequate,  consistent, 
descriptions  of  occupational  skill  requirements,  even  with  relatively  small  samples  of  incumbents 
within  each  occupation.  These  interrater  agreement  coefficients,  however,  are  dependent  on  the 
number  of  judges  available.  Thus,  it  is  useful  to  examine  the  single  judge  agreement  coefficients 
and  the  interrater  agreement  coefficients  that  would  have  been  obtained  if  30  judges  were 
available  for  each  occupation.  The  results  obtained  in  this  analysis  are  summarized  in  Table  3-2. 

As  may  be  seen  in  Table  3-2,  the  interrater  agreement  coefficients  that  could  be  expected 
if  30  incumbents  were  available  for  each  occupation  are  quite  large.  In  the  case  of  the  level  and 
importance  scales,  these  interrater  agreement  coefficients  would  be  in  the  mid  .90s.  Even  in  the 
case  of  the  job  entry  requirements  scale,  with  30  incumbents  per  occupation,  interrater  agreement 
coefficients  in  the  high  .80s  or  low  .90s  would  be  obtained 

The  good  reliability  of  incumbent  assessments  of  occupational  skill  requirements  is  also 
evident  in  the  single  rater  agreement  coefficients.  For  the  level  scale,  single  rater  agreement 
coefficients  were  in  the'  .40s,  while  the  importance  scale  yielded  single  rater  coefficients  in  the 
.30s,  although  lower  single  rater  agreement  coefficients,  typically  in  the  ,20s,  were  obtained  for 
the  job  entry  requirements  scale.  In  all  cases,  these  coefficients  were  of  a magnitude  indicating 
that  four  incumbents  should,  for  most  occupations,  be  sufficient  to  obtain  useable  data.  Fully 
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adequate  data  with  respect  to  requisite  interrater  agreement  coefficients  should  result  from 
samples  of  10  to  15  incumbents.  In  this  regard,  however,  it  should  be  noted  that  this  represents  a 
general  cross-occupation  conclusion  and  that  larger  samples  may  be  required  in  certain  somewhat 
more  heterogeneous  occupations. 

Scoring 

The  results  presented  above  speak  to  reliability  under  conditions  where  the  skills  are 
scored  using  the  full  scale.  In  other  words,  these  findings  apply  under  conditions  where  not 
relevant  responses  were  included  in  scoring  all  scales,  including  the  level  scale.  If  not  relevant 
was  endorsed,  the  level  was  scored  as  0,  the  importance  as  1 (not  important!,  and  the  job  entry 
requirement  as  2 (no,  it  can  be  learned  on  the  lob).  Although  we  view  this  scoring  as  the  most 
appropriate  method,  others  might  argue  for  different  methods  that  ignore  the  not  relevant 
response  option  or  use  that  response  differently  to  address  this  issue,  interrater  agreement 
coefficients  were  recalculated  for  each  scale,  dropping  not  relevant  responses.  Additionally,  the 

level  scale  agreement  coefficients  were  recalculated  using  a simple  relevant/not  relevant  coding 
scheme. 

The  interrater  agreement  coefficients  obtained  after  these  rescalings  are  presented  in 
Table  3-3.  The  interrater  agreement  coefficients  decreased  when  not  relevant  responses  were  not 
included  in  scoring.  The  decrease  averaged  .10  for  level,  .17  for  importance,  and  .19  for  job  entry 
requirement.  The  most  notable  differences,  found  across  all  three  scales,  occurred  for  the 
technology  skills,  which  had  much  lower  interrater  agreement  coefficients  when  not  relevant  was 
scored  as  missing,  instead  of  zero.  We  view  these  findings  as  supporting  the  intended  use  of  the 

not  relevant  response-that  is,  if  a skill  is  not  exercised  on  an  occupation,  it  is  not  important  and 
not  required  at  job  entry. 
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When  the  level  scale  was  dichotomized  to  relevant/not  relevant,  the  interrater  agreement 
coefficients  were  smaller  than  for  the  full  scale  scoring  procedure.  This  result  is  not  suiprising; 
more  noteworthy  is  the  finding  that  sizable  interrater  agreement  coefficients  lying  primarily  in 
the  .60s  or  .70s  were  obtained  with  the  use  of  this  dichotomous  scoring  procedure.  This  finding 
suggests  that  not  relevant  ratings  provided  some  unique  information  about  occupation 
characteristics  which  should  be  considered  in  scoring  the  skill  scales.  This  observation,  of  course, 
recommends  use  of  the  full  scale  scoring  procedure,  the  procedure  where  not  relevant  responses' 
are  used  as  the  lowest  scale  point.  This  scoring  system  was  applied  in  all  subsequent  analyses, 
except  those  reported  in  Table  3-15. 

Analyses  of  Variance 

Another  way  one  might  examine  interrater  agreement,  in  addition  to  the  simple  one-way 
analyses  described  above,  is  by  examining  interrater  agreement  under  conditions  where  the 
descriptions  of  skills  on  a given  scale  are  treated  as  a repeated  measures  variable.  The  results 
obtained  in  this  analysis  are  presented  in  Tables  3-4a,  3-4b,  and  3-4c,  which  present  the  variance 
attributable  to  each  of  the  relevant  sources  on  the  level,  importance,  and  job  entry  requirements 
scales,  respectively. 

Across  all  three  rating  scales,  significant  differences  were  observed  among  scores  for  a 
given  skill.  Apparently,  some  skills  are  viewed  as  being  more  central  to  incumbent  performance 
with  respect  to  level,  importance,  and  job  entry  requirements — a finding  consistent  with  our 
earlier  observations  about  the  need  for  basic,  problem  solving,  and  social  skills.  Even  taking  this 
general  trend  into  account,  however,  both  the  occupations  and  descriptor  by  occupations  terms 
had  significant  effects.  These  findings  indicate  that  the  skills,  regardless  of  the  scales  in  use,  can 
discriminate  occupations  and  that  some  skills  were  particularly  important  in  discriminating 
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certain  occupations  from  each  other.  Thus,  some  justification  is  provided  for  aggregating 
individual  data  to  describe  occupations. 

The  interrater  agreement  coefficients  resulting  from  these  analyses  are  presented  in  Table 
3-5,  along  with  the  single  rater  and  thirty  rater  agreement  coefficients  implied  by  these  analyses. 
As  with  our  earlier,  single  skill  analyses,  these  overall  analyses  indicate  that  the  skill  ratings- 
evidenced  adequate  interrater  agreement,  yielding  coefficients  of  .79,  .74,  and  .60  for  the  level, 
importance,  and  job  entry  requirements  scales,  respectively.  Again,  single  rater  agreement 
coefficients  were  good,  particularly  for  the  level  and  importance  ratings,  and  with  thirty  judges 
these  agreement  coefficients  would  lie  in  the  .90s.  Again,  however,  the  dichotomous  job  entry 
requirements  scale  provided  somewhat  weaker  results,  yielding  a thirty  judge  agreement 
coefficient  of  .83. 

The  hierarchical  structure  of  the  skills  permitted  higher-order  aggregate  scales  to  be 
formulated.  Thus,  one  might  examine  the  eight  discrete  problem  solving  skills  or  the  six  discrete 
social  skills  to  obtain  an  overall  estimate  of  problem  solving  skills  or  social  skills.  These 
aggregate  scales,  of  course,  might  prove  useful  in  drawing  general  conclusions  or  structuring  an 
information  search.  Accordingly,  these  interrater  agreement  analyses  were  rerun,  treating  scores 
on  aggregates  of  the  discrete  skill  scales  (simple  mean  scores  across  individual  scales  within  each 
aggregate)  specified  by  Mumford  and  Peterson  (1995)  as  variables.  Tables  3-6a,  3-6b,  and  3-6c 
present  the  results  obtained  in  these  analyses  for  the  level,  importance,  and  job  entry 
requirements  scales.  Table  3-7  presents  the  resulting  interrater  agreement  coefficients,  assuming 
a harmonic  mean  of  9.01  judges,  along  with  the  derivative  single  rater  and  thirty  rater  agreement 


coefficients. 
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In  all  of  these  analyses,  significant  effects  were  obtained  for  occupations,  the  aggregate 
scales,  and  the  occupation  by  aggregate  scale  interactions.  More  centrally,  these  theoretically- 
specified  aggregates  of  the  discrete  skills  evidenced  good  reliability,  with  coefficients  of  .86,  .86, 
and  .69  being  obtained  for  the  level,  importance,  and  job  entry  requirement  scales,  respectively. 
These  aggregates,  as  expected  due  to  the  covariation  among  skills,  yielded  higher  observed 
agreement  coefficients.  In  fact,  the  single  rater  agreement  coefficients  for  these  scale  aggregates 
were  .40,  .40,  and  .20  for  the  level,  importance,  and  job  entry  requirements  scales,  while  the 
thirty  judge  coefficients  were  .95,  .95,  and  .88,  respectively.  Thus,  these  aggregates,  like  the 
discrete  skill  scales  they  were  developed  from,  evidenced  adequate  reliability. 

Descriptor  and  Scale  Relationships 

Although  the  skill  ratings  appear  to  be  reliable,  little  has  been  said  about  the  relationships 
among  the  rating  scales.  Table  3-8  presents  the  average,  across  skills  correlations  among  the 
level,  importance,  and  job  entry  requirement  ratings.  As  can  be  seen,  the  level  and  importance 
scales  yielded  a sizable  positive  correlation  of  .95.  When  this  finding  is  considered,  along  with 
the  relatively  small  standard  deviation  obtained  in  this  analysis,  it  seems  reasonable  to  conclude 
that  the  importance  scale  added  little  to  the  level  scale  ratings. 

The  correlations  of  the  level  and  importance  scales  with  the  job  entry  requirements  scale 
are  also  presented  in  Table  3-8.  Bearing  in  mind  the  scaling  of  the  job  entry  requirements  ratings, 
where  lower  ratings  (one  on  a two-point  scale)  indicated  that  the  skill  was  required  at  the  time  of 
job  entry,  it  is  not  surprising  that  negative  relationships  were  obtained  in  this  analysis.  This  scale 
yielded  sizable  negative  correlations  of  -.71,  and  -.74  with  the  level  and  importance  ratings, 
respectively.  When  these  relatively  smaller  correlations  are  considered  along  with  the  larger 
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standard  deviations  obtained  for  the  scale,  they  suggest  that  the  job  entry  requirements  ratings 
might  provide  some  unique  descriptive  information. 

In  evaluating  the  relationships  among  ratings  of  job  entry  requirements  and  the  level  and 
importance  ratings,  another  point  must  be  borne  in  mind:  the  job  entry  requirements  ratings  were 
substantially  less  reliable  than  the  level  and  importance  ratings.  Because  these  findings  with 
regard  to  the  job  .entry  requirements  scales  may  reflect  an  artifact  associated  with  reliability  and 
because  the  level  and  importance  ratings  were  strongly  related,  further  discussion  of  the  findings 

for  the  skill  scales  will  focus  on  the  level  ratings.  However,  as  necessary,  the  data  for  the  other 
scales  will  be  presented. 

Tables  3-9a,  3-9b,  and  3-9c  present  the  correlations  among  the  level,  importance,  and  job 
entry  requirements  ratings  at  the  occupation  level.  Tables  3-lOa,  3- 10b,  and  3-10c  present  the 
same  correlations  but  at  the  individual  level  using  four  individuals  per  occupation.  Because  our 
primary  concern  of  the  present  study  is  occupation  description,  we  will  focus  on  the  correlations 
obtained  at  the  occupation  level-particularly,  the  level  ratings,  which  appeared  to  provide  the 
most  unique  information. 

In  examining  the  correlations  among  skills  on  the  level  scale,  it  is  clear  that  the  skills 
evidenced  an  interpretable,  substantively  meaningful  pattern  of  relationships.  For  example, 
ratings  of  the  Mathematics  (Descriptor  #5)  and  Science  (Descriptor  #6)  skills  evidenced  the 
expected  positive  relationship  (r  = .56).  These  skills,  however,  were  only  weakly  related  to  Social 
Perceptiveness  (Descriptor  #1 1),  yielding  correlations  of  .09  and  .26  for  Mathematics  (Descriptor 
#5)  and  Science  (Descriptor  #6),  respectively.  Along  similar  lines,  learning  to  learn  skills,  such  as 
Learning  Strategies  (Descriptor  #9),  yielded  the  expected  positive  correlations  with  Operations 
Analysis  (Descriptor  #25)  (r  = .49)  and  Technology  Design  (Descriptor  #26)  (r  = .42),  but  weaker 
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relationships  with  more  routine  technical  skills,  such  as  Operation  Monitoring  (Descriptor  #31)  (r 
= .10)  and  Operation  and  Control  (Descriptor  #32)  (r  = .06). 

Other  examples  of  this  sort  might  be  cited.  However,  the  examples  presented  above 
suffice  to  indicate  that  the  skill  scales  evidence  a meaningful  pattern  of  relationships.  In  this 
regard,  however,  it  is  important  to  note  three  broader  trends  in  the  pattern  of  the  relationships 
observed  among  level  requirement  ratings  on  the  skill  scales.  First,  these  skills  typically 
evidenced  sizable  positive  relationships,  suggesting  that  a job  requiring  one  skill  is  likely  to 
require  a number  of  other  skills.  Second,  somewhat  stronger  relationships  were  observed  among 
skills  within  categories  as  opposed  to  across  categories.  Thus,  the  problem  solving  skills  were 
more  strongly  related  to  each  other  than  to  the  social  skills.  Third,  and  finally,  the  technical  skills 
tended  to  produce  the  weakest  correlations  with  skills  drawn  from  other  categories~a  finding 
which  suggests  that  technical  skills  may  reflect  a relatively  distinct  set  of  occupational  skill 
requirements. 

Factor  Structure 

The  correlations  observed  among  the  skills  brings  to  the  fore  a new  question:  do  those 
correlations,  overall,  evidence  a meaningful  pattern  of  interrelationships?  To  address  this  issue,  a 
principal  components  factoring  was  conducted  using  the  correlations  obtained  at  the  job  level. 
Table  3-11  summarizes  the  results  obtained  in  this  analysis,  following  a varimax  rotation. 

Inspection  of  the  eigenvalues  and  a scree  plot  indicated  that  a three  factor  solution 
provided  the  most  plausible  structure  for  summarizing  the  relationships  observed  among  the  skill 
scales.  These  factors  accounted  for  8 1 % of  the  total  variance  for  ratings  on  the  level  scale.  The 
communalities  indicated  that  these  factors  also  accounted  for  most  of  the  variance  in  ratings  of 
the  various  basic  and  cross-functional  skills. 
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The  first  factor  extracted  in  this  analysis,  accounting  for  38%  of  this  variance,  was  labeled 
cognitive  skills.  As  might  be  expected,  all  of  the  basic  and  problem  solving  skills  yielded  sizable 
loadings  on  this  factor.  For  example.  Writing  (Descriptor  #3)  (r=  .89),  Synthesis/Reorganization 
(Descriptor  #20)  (r  = .89),  Critical  Thinking  (Descriptor  #7)  (r  = .85),  Speaking  (Descriptor  #4)  (r 
= .85),  Problem  Identification  (Descriptor  #17)  (r  = .83)  and  Information  Gathering  (Descriptor 
#18)  (r  = .89)  all  produced  sizable  loadings.  It  was  also  found,  however,  that  most  of  the  social 
skills  provided  sizable  loadings,  including  Coordination  (Descriptor  #12)  (r  = .80),  Persuasion 
(Descriptor  #13)  (r  = .80),  and  Instructing  (Descriptor  #15)  (r  = .74).  This  pattern  of  loadings, 
although  at  first  glance  surprising,  is  consistent  with  the  observations  of  Carroll  (1993)  and 

Mumford  and  Peterson  (1995),  who  note  that  interactional  skills  often  depend  on  broader 
cognitive  capacities. 

The  second  factor  extracted  in  this  analysis,  accounting  for  25%  of  the  total  variance,  was 
labeled  technical  skills.  As  might  be  expected,  virtually  all  of  the  technical  skills,  including 
Troubleshooting  (Descriptor  #35)  (r  = .96),  Testing  (Descriptor  #30)  (r  = .94),  Operation 
Monitoring  (Descriptor  #3 1)  (r  = .88),  Equipment  Maintenance  (Descriptor  #34)  (r  = .87),  and 
Technology  Design  (Descriptor  #26)  (r  = .82),  had  high  loadings  on  this  factor.  Science 

(Descriptor  #6)  (r  = .70)  and  Mathematics  (Descriptor  #5)  (r  = .54)  also  yielded  sizable  loadings 
on  this  factor. 

The  third,  and  final,  factor  extracted  in  this  analysis  accounted  for  17%  of  the  variance  in 
level  ratings  on  the  skill  scales.  This  factor,  labeled  organizational  skills,  was  defined  in  terms  of 
the  systems  skills,  including  Visioning  (Descriptor  #37)  (r  = .71),  Identification  of  Downstream 
Consequences  (Descriptor  #39)  (r  = .78),  and  Judgment  and  Decision  Making  (Descriptor  #41)  (r 
= .56),  as  well  as  the  four  resource 


management  skills—Management  of  Personnel  Resources 
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(Descriptor  #46)  (r  = .83),  Management  of  Financial  Resources  (Descriptor  #44)  (r  = .81), 
Management  of  Material  Resources  (Descriptor  #45)  (r  = .80),  and  Time  Management 
(Descriptor  #43)  (r  = .52).  In  addition,  certain  social  skills,  particularly  those  that  would 
influence  getting  things  done  in  complex  social  systems,  such  as  Negotiation  (Descriptor  #14)  (r 
= .68)  and  Persuasion  (Descriptor  #13)  (r  = .46),  also  yielded  sizable  loadings  on  this  factor. 

Taken  as  a whole,  the  results  obtained  in  this  analysis  provide  some  initial  evidence  for 
the  meaningfulness  of  the  occupational  skill  ratings.  It  is  hardly  surprising  that  occupational 
skills  could  be  organized  in  terms  of  cognitive,  technical,  and  organizational  skills.  In  this  regard, 
however,  a word  of  caution  is  in  order.  Those  factors  indeed  tell  us  something  about  relationships 
among  the  skill  scales.  However,  they  have  little  to  say  about  how  these  skills  differentiate 
occupations.  Thus,  it  should  not  be  assumed  that  these  factors  necessarily  provide  an  adequate 
summarization  system  when  our  concern  is  describing  the  similarities  and  differences  among 
occupations. 

Future  analyses  of  0*NET  data  will  likely  include  confirmatory  factor  analyses 
investigating  both  the  robustness  of  the  exploratory  model  described  above,  and  the  strength  of 
the  theoretical  hierarchical  relationship  among  the  skills  described  by  Mumford  and  Peterson 
(1995).  However,  the  currently-available  data  are  not  sufficient  to  support  either  analysis. 

Occupation  Differences 

Some  initial  evidence  bearing  on  the  ability  of  the  skill  scales  to  capture  the  similarities 
and  differences  among  occupations  might  be  obtained  by  contrasting  the  mean  profiles  of 
occupations  on  the  various  skill  scales.  Table  3- 12a  presents  the  means  and  standard  deviations 
of  level  scale  ratings  on  six  occupations  selected  to  reflect  distinct  types  of  employment:  (1) 
General  Managers  and  Top  Executives,  (2)  Computer  Programmers,  (3)  Registered  Nurses,  (4) 
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Police  Patrol  Officers,  (5)  Janitors  and  Cleaners,  and  (6)  Maintenance  Repairers,  General  Utility. 
Table  3-12b  summarizes  the  importance  scale  ratings. 

The  mean  scores  of  incumbents  on  these  skill  scales,  of  course,  reflect  the  general  trends 
noted  m our  earlier  discussion  of  these  overall  descriptive  data.  For  example,  across  occupations, 
problem  solving  and  social  skills  typically  received  high  ratings  with  respect  to  level 
requirements.  Even  bearing  these  general  trends  in  mind,  however,  a review  of  mean  ratings  for 
these  occupations  does  indicate  that  they  evidenced  an  interpretable  pattern  of  differences  on  the 
level  scale.  For  example.  Computer  Programmers  indicated  the  highest  level  of  Programming 
(Descriptor  #29)  (M  = 6.40,  SD  = .89)  of  all  the  occupations  under  consideration.  On  the  other 
hand.  Programmers  tended  to  indicate  that  their  jobs  did  not  require  a high  level  of  Social 
Perceptiveness  (Descriptor^  1)  (M  = 2.60,  SD  = .89).  As  might  be  expected.  General  Managers 
and  Top  Executives  had  the  highest  level  ratings  on  the  four  resource  management  skills  vis-a- 
vis  the  five  other  occupations  under  consideration:  Time  Management  (Descriptor  #43)  (M  = 
5.42,  SD  =1.16),  Management  of  Financial  Resources  (Descriptor  #44)  (M  = 4.43,  SD  = 1.76) 
Management  of  Material  Resources  (Descriptor  #45)  (M  = 4.50,  SD  = 1.80),  and  Management  of 
Personnel  Resources  (Descriptor  #46)  (M  = 5.44,  SD  = 1 .44).  Moreover,  Police  Patrol  Officers, 
consistent  with  the  nature  of  the  Patrol  Officers' jobs,  had  unusually  low  scores  on  Management 
of  Financial  Resources  (Descriptor  #44)  (U  = .90,  SD  = 1 .64).  Police  Officers,  however,  did 
need  the  highest  level  of  Active  Listening  (Descriptor  #2)  (M  = 5.86,  SD  = .74)  and  Negotiation 
(Descriptor  #14)  (M  = 5.14,  SD=  1.28)  skills. 

As  might  be  expected.  Janitors  and  Cleaners  typically  had  the  lowest  reported  skill 
requirements  for  all  of  the  skill  scales.  In  this  regard,  however,  it  is  important  to  note  that  Janitors 
and  Cleaners  did  have  somewhat  higher  scores  on  the  Equipment  Maintenance  (Descriptor  #34) 
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(M  = 3.17,  SD  = 2.41),  Troubleshooting  (Descriptor  #35)  (M  = 2.96,  SD  = 2.55),  and  Repairing 
(Descriptor  #36)  (M  = 2.65,  SD  = 2.39)  scales,  a finding  consistent  with  the  nature  of  janitorial 
jobs.  As  might  be  expected.  Maintenance  Repair  Workers  had  substantially  higher  scores  on 
Equipment  Maintenance  (Descriptor  #34)  (M  = 5.01,  SD-1.38),  Troubleshooting  (Descriptor 
#35)  (M  = 4.96,  SD  = 1.29),  and  Repairing  (Descriptor  #36)  (M  = 5.04,  SD  = 1 .34)  than  did 
Janitors  and  Cleaners. 

Although  other  examples  of  this  sort  might  be  cited,  the  evidence  presented  above  does 
lead  to  a noteworthy  conclusion.  Specifically,  it  appears  that  the  skill  scales  provide  a meaningful 
description  of  the  similarities  and  differences  among  occupations.  This  point  is  of  some 
importance  since  ultimately  a viable  descriptive  system  must  be  capable  of  capturing  and 
accurately  reflecting  meaningful  differences  in  occupational  requirements. 

Discriminant  Analyses 

Our  foregoing  analysis  of  mean  differences  across  occupations  was,  of  course,  based  on  a 
limited  set  of  well-known  occupations.  This  analysis,  however  valuable,  begs  a further  question. 
How  well  do  level  ratings  on  the  skill  scales  differentiate  occupations  in  a larger  sample  of 
occupations?  To  address  this  issue,  a discriminant  analysis  was  conducted.  The  results  obtained 
in  this.analysis  are  presented  in  Table  3-13,  which  displays  both  the  loadings  on  the  discriminant 
functions  and  the  discriminating  variance  attributable  to  a given  skill. 

The  discriminant  analysis  provided  six  functions  which  appeared  to  be  interpretable  and 
had  nontrivial  eigenvalues.  These  functions  and  the  skill  variables  they  were  derived  from  appear 
to  provide  a useful  mechanism  for  discriminating  occupations.  If  all  functions,  including  these 
six,  are  used,  62%  of  the  incumbents  are  assigned  to  the  occupations  from  which  they  were 
drawn.  If  incumbents  were  evenly  distributed  across  the  35  occupations  in  this  analysis,  then  the 
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expected  or  base  rate  of  correct  classification  would  be  about  3%  for  random  assignment.  In  this 
sample,  incumbents  are  not  evenly  distributed  across  occupations,  but,  even  if  we  double  their 
expected  rate  to  6%,  the  observed  number  is  still  substantially  higher,  indicating  that  patterns  of 
scores  on  the  level  scale  are  indeed  an  effective  vehicle  for  capturing  the  similarities  and 
differences  among  occupations  across  a range  of  occupations. 

The  results  obtained  in  this  discriminant  analysis  also  indicate  that  all  of  the  skills  made 
some  contribution  to  discriminating  the  occupations  from  each  other.  Table  3-13  presents  the 
sum  of  the  squared  function  coefficients,  as  well  as  the  T|2  coefficients  reflecting  the  variance  in 
job  assignments  accounted  for  by  ratings  on  each  of  the  skill  scales.  The  squared  function 
coefficients  ranged  between  .72  and  .10  and  the  T|2  coefficients  between  .41  and  .18..  The  best 
discriminators  of  job  assignments  were  apparently  the  technical  and  scientific  skills,  particularly 
Repairing  (Descriptor  #36),  Installation  (Descriptor  #28),  and  Operation  Monitoring  (Descriptor 
#31)— skills  characterizing  production  jobs.  The  Learning  Strategies  (Descriptor  #9),  Instructing 
(Descriptor  #15),  and  Service  Orientation  (Descriptor  #16 ) skills  produced  the  weakest 
discrimination.  In  this  regard,  however,  it  should  be  recognized  the  T|2  and  function  coefficients 
obtained  for  these  skills  indicated  that  they  still  made  nontrivial  contributions  to  discrimination. 

In  addition  to  indicating  that  all  of  the  skills  contributed  to  discrimination,  the  findings 
obtained  in  this  analysis  provide  some  clues  about  how  skills  discriminate  occupations. 
Inspection  of  the  skills  loading  on  the  six  functions  retained  in  this  analysis  indicated  that  the 
functions  appeared  to  reflect,  respectively:  (1)  technical  repair.  (2)  systems  direction.  (3)  machine 
production,  (4)  research  and  development.  (5)  interactive  problem  solving,  and  (6)  computer 
applications . The  nature  of  these  functions,  as  defined  by  the  function  loadings,  suggests  that  the 
skills  discriminate  occupations  by  capturing  integrated  patterns  of  skills  related  to  core  job 
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functions.  However,  these  particular  findings  are  limited  in  generality,  since  we  have  just  35 
occupations  in  our  sample.  A broader  group  of  occupations  might  indicate  different  skills  as  the 
best  discriminators. 

Convergence  With  Analysts’  Ratings 

These  discriminant  analysis  findings,  of  course,  provide  some  important  evidence  for  the 
meaningfulness,  or  validity,  of  the  descriptive  information  provided  by  the  skill  scales.  Some 
further  evidence  for  the  meaningfulness  of  the  descriptive  information  provided  by  these  scales 
may  be  obtained  by  contrasting  incumbents’  and  analysts’  ratings  on  the  skill  level  scales.  The 
results  of  this  comparison  are  summarized  in  Table  3-  14a,  which  provides  the  means,  standard 
deviations,  and  interrater  agreement  coefficients  obtained  for  incumbents’  and  analysts’  ratings 
on  each  skill  for  the  level  scale.  (Table  3- 14b  presents  the  importance  ratings).  Table  3- 14a  also 
presents  the  t and  F tests  comparing  the  means  and  standard  deviations  of  incumbents’  and 
analysts’  ratings,  along  with  the  correlation  coefficient  and  d2  index  contrasting  incumbents'  and 
analysts  ratings  of  occupational  skill  requirements.  The  analysts’  and  incumbents’  ratings  both 
provided  sizable  interrater  agreement  coefficients  for  ratings  of  the  level  of  skill  requirements 
The  interrater  agreement  coefficients  obtained  in  the  analyst  sample,  however,  were  typically 
somewhat  larger  than  those  obtained  in  the  incumbent  sample.  Overall,  the  analyst  agreement 
coefficients  were  .10  points  higher  than  the  incumbents’  ratings,  resulting  in  agreement 
coefficients  in  the  mid  .90s  as  opposed  to  the  mid  .80s. 

More  centrally,  however,  both  incumbent  and  analysts  ratings  of  occupational  skill 
requirements  displayed  good  agreement.  The  median  correlation  observed  between  the 
incumbents  and  analysts’  ratings  was  .73.  Thus,  analysts  and  incumbents  displayed  substantial 
agreement  in  their  descriptions  of  the  patterns  of  occupational  skill  requirements.  This  finding,  of 
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course,  provides  some  compelling  evidence  for  the  convergent  validity  of  ratings  of  occupations' 
skill  requirements.  Along  similar  lines,  the  F test  indicated  that  variances  of  incumbents'  and 
analysts’  mean  ratings  of  skill  requirements  across  occupations  were  also  quite  similar,  with  only 
5 of  the  46  F tests  yielding  significant  differences  at  the  .05  level. 

Although  incumbents  and  analysts  displayed  substantial  agreement  in  the  patterns  of 
otcupational  skill  ratings,  differences  were  generally  observed  between  incumbents  and  analysts 
with  respect  to  mean  level  ratings.  Typically,  analysts’  ratings  on  the  basic,  problem  solving, 
social,  systems,  and  resource  management  skills  were  lower  than  incumbents'  ratings.  This 
finding  suggests  that  incumbents  may  see  higher  levels  of  skill  requirements  than  analysts.  It 
should  be  noted,  however,  that  this  trend  was  not  as  pronounced  for  the  technical  skills,  where  in 
some  cases  analysts  indicated  higher  levels  of  skill  requirements  than  incumbents.  Thus,  it 
appears  that  analysts'  and  incumbents'  ratings  may  reflect  distinct  perspectives-in  the  case  of 
analysts,  one  based  on  cross-occupation  comparisons,  and  in  the  case  of  incumbents,  one  based 
on  the  more  salient  aspects  of  work  on  the  occupation  at  hand.  The  d2  results  in  Table  3-14a 
indicate  that  the  absolute  sizes  of  these  differences  are  less  than  one  and  one-half  scale  points 

(the  square  root  of  d2),  with  the  exception  of  some  of  the  basic,  social,  and  resource  management 
skills. 

Table  3-15  displays  analogous  results,  but  based  on  a dichotomous  rescoring  of  the  data 
as  relevant/not  relevant.  In  general,  analysts  were  more  likely  to  rate  skills  as  relevant  than  were 
incumbents.  This  is  an  interesting  finding,  particularly  in  light  of  the  above  finding  that,  overall, 
level  ratings  tended  to  be  lower  for  analysts.  As  we  also  saw  in  Table  3-3,  the  reliabilities  are 
predictably  lower,  as  a result  of  the  dichotomization  of  the  scale.  In  fact,  for  Operation  and 


Skins  3-23 


Control  (Descriptor  #32),  which  had  the  lowest  reliability  when  scored  on  the  full  scale,  the 
reliability  for  the  analysts’  ratings  scored  dichotomously  is  zero. 

Table  3-16  reports  the  results  for  a principle  components  analysis  of  the  analysts’  mean 
level  ratings  for  the  35  occupations.  The  analysts’  ratings  were  found  to  support  only  two  factors, 
in  contrast  to  the  three  factors  yielded  in  the  analysis  of  the  incumbents’  ratings,  presented  in 
Table  3-11  and  described  in  detail  above.  The  second  of  these  is  very  similar  in  loading  patterns 
to  the  technical  skills  factor  in  the  analysis  of  incumbents’  ratings.  The  first,  however,  is 
essentially  a combination  of  the  first  and  third  factors-cognitive  skills  and  organizational  skills— 
in  that  analysis.  Morgeson  and  Campion  (1996)  have  suggested  that  raters  with  limited 
knowledge  of  an  occupation  may  rely  on  overall  impressions  of  an  occupation  in  making  their 
ratings,  possibly  causing  their  ratings  to  be  more  internally  consistent  than  those  of  job 
incumbents  with  their  greater  job  knowledge.  If  occupational  analysts,  in  relying  on  a list  of 
occupation-specific  tasks  for  their  knowledge  of  an  occupation,  tend  to  rate  skills  generally  high 
or  low  based  on  a general  impression  of  the  occupation,  then  their  ratings  would  indeed  be  likely 
to  yield  fewer  factors,  as  was  found  here. 

Additional  Validity  Evidence 

Convergence  is,  of  course,  only  one  characteristic  of  meaningful  descriptive  systems.  As 
Messick  (1995)  points  out,  valid  descriptive  systems  should  also  allow  us  to  draw  meaningful 
inferences  about  the  objects  of  interest— in  this  case,  occupations.  Recently,  in  the  popular  press, 
we  have  heard  a number  of  discussions  about  the  relationship  between  skills  and  employment 
patterns.  Neither  time  nor  the  nature  of  the  available  data  allows  us  to  explore  all  of  the 
hypotheses  formulated  in  the  course  of  this  debate.  Two  questions,  however,  seem  of  particular 
interest  with  regard  to  the  skill  measures.  First,  are  formal  educational  requirements  related  to 
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higher  skill  levels?  Second,  are  skill  requirements  in  high  performance  organizations  different 
than  in  more  traditional  organizations? 

To  address  our  question  about  educational  requirements,  the  measures  described  in  the 
Training,  Education,  Licensure,  and  Experience  chapter.  Chapter  5,  were  used.  Here,  a measure 
of  overall  educational  requirements  was  developed  by  determining  the  level  of  education  • 
required.  The  majority  of  incumbents  in  19  of  the  35  occupations  reported  that  their  jobs  required 
a high  school  diploma  (or  high  school  equivalence  certificate)  or  less.  The  majority  of 
incumbents  in  the  remaining  16  occupations  reported  that  their  jobs  required  at  least  some  post- 
secondary training  (e.g.,  vocational  training,  college  courses,  or  college  or  graduate  degrees).  The 
means  and  standard  deviations  of  occupational  mean  ratings  on  the  level  scale  in  these  two 
groups  were  contrasted  using  at  test  and  an  F test.  Table  3-17a  summarizes  the  results  obtained 
in  this  analysis. 

In  accordance  with  the  hypothesis  that  increased  educational  requirements  are  related  to 
higher  skills,  it  was  found  that  scores  on  the  level  scales  were  generally  higher  for  occupations 
with  higher  educational  requirements  (Mm  = 3.53,  MSD  = 1.16)  than  for  occupations  with  lower 
educational  requirements  (Mm  = 2.20,  MSD  = 0-89).  Thus,  overall,  our  findings  seem  to  confirm 
the  conventional  wisdom.  In  this  regard,  however,  it  should  be  noted  that  these  differences  were 
more  pronounced  with  regard  to  basic  skills  (Descriptor  #l-Descriptor  #1 1)  (Mm  = 4.32,  MSD  = 
0.79  vs.  Mm  = 2.97,  Msd  = 0.79)  and  problem  solving  skills  (Descriptor  # 17-Descriptor  #24)  (Mm 
= 4.26,  Msd  = 0.9 1 vs.  Mm  = 2.58,  Msd  = 0.82),  than  technical  skills  (Descriptor  #25-Descriptor 
#36)  (Mm  = 2.38,  Msd  = 1-64  vs.  Mm  = 1-26,  Msd  = 1-06). 

To  provide  some  data  that  might  be  used  to  answer  our  second  question,  we  drew  from 
the  analyses  presented  in  the  chapter  on  organizational  context.  Chapter  8.  In  that  chapter,  the 
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characteristics  of  the  establishments  employing  incumbents  are  assessed.  A hierarchical  factoring 
of  the  organizational  context  items  administered  to  organizational  representatives  resulted  in  a 
coherent  factor  structure  where  factors  (use  of  teams  and  organizational  goal  setting,  for 
example)  were  related  to  the  known  characteristics  of  high  performance  organizations.  Four 
factor  scores  were  used  in  this  analysis-use  of  teams,  information  sharing,  organizational  goal 
setting,  and  high  performance  human  resources  practices.  This  selection  was  based  on  these 
factors’  high  loadings  on  a high  performance  practices  second-order  factor  in  the  analysis 
reported  in  Table  8-23  in  Chapter  8,  combined  with  their  theoretical  identification  as 
organizational  characteristics  related  to  high  performance.  The  scores  of  incumbent 
establishments  on  these  factors  were  obtained,  standardized,  and  summed,  and  those 
establishments  in  the  upper  quartile  on  the  composite  were  assigned  to  the  high  performance 
group,  while  those  with  scores  in  the  lower  quartile  were  assigned  to  the  traditional  (not  high 
performance)  group. 

Two  occupations  were  selected  for  this  analysis:  (1)  General  Office  Clerks,  and  (2)  First 
Line  Supervisors,  Clerical/Administrative.  For  each  of  these  occupations,  at  least  15  incumbents 
from  high  performance  establishments  and  15  from  traditional  establishments  provided  Skills 
ratings.  Most  of  the  other  sampled  occupations  did  not  have  adequate  numbers  of  incumbents  in 
each  of  these  groups  to  permit  this  analysis.  We  anticipate  that,  as  the  0*NET  database  grows, 
we  will  be  able  to  repeat  this  analysis  for  more  occupations,  and  with  more  incumbents  for  these 
occupations. 

The  results  of  these  analyses  are  presented  in  Tables  3- 17b  and  3- 17c.  Generally,  there 
are  few  statistically  significant  differences— not  a surprising  finding  given  the  relatively  low 
numbers  of  respondents  in  each  group,  and  the  fact  that  these  are  within-occupation  comparisons. 

O 
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Perhaps  the  most  clear-cut  conclusion  to  emerge  from  this  analysis  is  that  variances  of  skill  level 
ratings  were  generally  higher  for  incumbents  in  high  performance  organizations,  suggesting  that 
such  organizations  may  be  allowing  employees  to  develop  and  express  their  skills  in 
idiosyncratic  ways. 

Additionally,  it  seems  that  while  some  lower-level  jobs  in  high  performance 
organizations  may  require  higher  skill  levels,  supervisory  jobs  in  high  performance  organizations 
may  actually  require  lower  skill  levels.  For  General  Office  Clerks,  across  all  skills,  incumbents 
working  in  high  performance  organizations  reported  slightly  higher  skill  requirements  (Mm  = 
2.11,  Mst)  = 1.78)  than  incumbents  working  in  more  traditional  organizations  (Mm  = 1-68,  = 

1.55).  These  differences,  however,  were  especially  large  with  regard  to  the  problem  solving  skills 
(Descriptor  # 17-Descriptor  #24)  (Mm  = 2.81,  MSD  = 1.83  vs.  Mm  = 2.24,  = 1.60),  social  skills 

(Descriptor  #1 1-Descriptor  #16)  (Mm  = 2.83,  Msd  = 2.02  vs.  Mm  = 2.04,  M®  = 1.81),  and  resource 
management  skills  (Descriptor  #43-Descriptor  #46)  (Mm  = 1.87,  Mso  = 2.06  vs.  Mm  = 1.21,  MSD  = 
1.49).  This  finding  is  consistent  with  the  goal  of  many  high  performance  organizations  to 
encourage  employees  to  manage  their  own  work. 

In  contrast,  for  First  Line  Supervisors,  those  working  in  high  performance  organizations 
reported  lower  skill  requirements  than  those  working  in  traditional  organizations  (Mm  = 3-30.  MSD 
= 1.71  vs.  Mm  = 3.91,  Msd  = 1-57).  These  differences  were  especially  large  with  regard  to  the 
problem  solving  skills  (Descriptor  # 17-Descriptor  #24)  (M«  = 3.89,  MSD  = 1-65  vs.  Mm  = 4-85. 
Msd  = 1-22),  social  skills  (Descriptor  # 1 1-Descriptor  #16)  (Mm  = 4.08,  MsD  = 1-83  vs.  = 4-75. 
Msd  = 1-07),  systems  skills  (Descriptor  #37-Descriptor  #42)  (Mm  = 3.29,  Msd  = 1-93  vs.  Mm  = 
4.39,  Msd  = 1 -73),  and  resource  management  skills  (Descriptor  #43-Descriptor  #46)  (Mm  = 4-02, 
Msd  = 1-78  vs.  Mm  = 4.60,  MSd  = 1-87).  It  appears,  then,  that  at  least  some  low-level  supervisory 
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jobs  in  high  performance  organizations  have  lower  skill  demands  than  similar  jobs  in  traditional 
organizations.  This  may  be  related  to  the  above  finding  that  lower  level  employees  in  high 
performance  organizations  have  higher  levels  of  skills— perhaps  such  employees  require  less 
supervision.  This  issue  merits  further  investigation  when  additional  data  are  available. 

Conclusions 

Having  evaluated  the  meaningfulness  of  the  skill  scales  using  the  data  gathered  in  this 
prototype  study,  it  would  seem  germane  to  consider  some  of  the  broader  methodological  and 
substantive  conclusions  flowing  from  these  findings.  Before  turning  to  these  broader  implications 
of  the  present  study,  certain  limitations  of  the  present  study  should  be  noted.  To  begin,  although 
our  findings  are  based  on  a sample  of  the  more  populous  occupations  found  in  the  American 
economy,  it  is  certainly  true  that  we  have  examined  only  a small  number  of  occupations. 
Accordingly,  some  caution  is  called  for  in  generalizing  our  findings  to  all  jobs  in  the  economy. 

Along  similar  lines,  it  must  be  recognized  that  despite  the  systematic  random  sampling 
procedures  in  use,  our  findings  cannot  be  viewed  as  providing  definitive  normative  Hata  about 
occupational  skill  requirements.  In  many  cases,  only  a relatively  small  number  of  incumbents— 
four  or  five— provided  descriptive  data  bearing  on  occupational  skill  requirements.  Overall,  about 
18  incumbents  on  average  were  available  for  the.  35  occupations.  Samples  of  this  size  may  be 
sufficient  to  obtain  adequate  agreement  with  respect  to  the  description  of  occupational  skill 
requirements.  Nevertheless,  they  should  not  be  viewed  as  providing  a truly  normative  description 
of  jobs  employing  tens  of  thousands  of  workers. 

In  addition  to  these  concerns  about  the  number  of  occupations  and  the  number  of 
incumbents  per  occupation,  three  other  limitations  of  the  present  study  need  to  be  borne  in  mind. 
The.  first  of  these  limitations  pertains  to  the  nature  and  range  of  the  evidence  bearing  on  the 
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meaningfulness  of  the  descriptive  information  provided  by  the  skill  scales.  In  the  present  study, 
evidence  bearing  on  the  meaningfulness  of  the  descriptive  information  provided  by  these  scales 
was  derived  primarily  through  analyses  examining  internal  validity.  This  evidence,  however 
impressive,  cannot  tell  us  whether  these  scales  will  evidence  meaningful  relationships  with  other 
measures  of  occupational  requirements.  This  external  validity  evidence,  at  a minimum  requires 
cross-domain  analyses  beyond  the  scope  of  the  present  effort.  Some  relevant  cross-domain 
analyses  are  reported  in  Chapter  15. 

The  second  limitation  inherent  in  these  analyses  must  be  recognized  by  any  effort 
intended  to  provide  evidence  for  the  meaningfulness  of  a new  set  of  measures.  Evidence  for  the 
meaningfulness  of  a set  of  measures  can  be  obtained  using  a variety  of  techniques.  Although  a 
number  of  analyses  were  used  to  accrue  evidence  for  the  meaningfulness  of  our  skill  measures, 
we  could  not,  and  have  not,  conducted  every  analysis  that  might  be  used  to  provide  evidence 
bearing  on  the  meaningfulness  of  these  measures.  As  a result,  any  conclusions  flowing  from 
these  findings  must  be  viewed  as  contingent  and  subject  to  change  with  future  research. 

Third,  and  finally,  it  should  be  noted  that  any  evidence  bearing  on  the  meaningfulness  of 
a measure  is  referenced  against  a particular  application  of  the  measure.  In  the  present  study,  our 
intent  was  to  describe  occupational  skill  requirements.  Accordingly,  our  findings  must  be 
assessed  with  this  point  in  mind  and  some  caution  is  called  for  in  extrapolating  our  findings  to 
other  potential  uses  of  skill  measures.  For  example,  our  findings  have  little  to  say  in  any  direct 
sense  about  the  procedures  that  should  be  used  to  develop  occupational  skills  nor  can  they  be 
used  to  draw  explicit  conclusions  about  the  kind  of  procedures  that  should  be  used  to  assess 
workers  skills.  Instead,  our  findings  speak  primarily  to  the  measurement,  structure,  and  nature  of 
occupational  skill  requirements. 
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Methodological  Conclusions 

Even  bearing  these  caveats  in  mind,  we  do  believe  that  the  findings  obtained  in  the 
present  study  have  some  important  implications  for  the  assessment  of  occupational  skill 
requirements.  To  begin,  it  appears  that  it  is  possible  to  formulate  rating  scales  that  can  be  used  to 
obtain  reliable,  meaningful  assessments  of  occupational  skill  requirements  from  either 


incumbents  or  analysts.  The  apparent  feasibility  of  assessing  occupational  skill  requirements 


using  a rating  strategy  is  noteworthy  because  it  represents  a relatively  low-cost  procedure  for 
establishing  occupational  skill  requirements. 


In  this  regard,  however,  it  seems  necessary  to  note  the  important  characteristics  of  the 
procedures  used  to  obtain  these  ratings.  First,  the  skills  were  defined  in  simple  operational  terms 
which  were  intended  to  be  understood  by  job  incumbents  with  about  a sixth  grade  reading  level. 
Second,  the  level  scale  provided  a set  of  concrete  behavioral  illustrations  of  the  skill  under 
consideration.  It  is  therefore  quite  possible  that  similar  results  would  not  be  obtained  using 
different  definitions  or  different  rating  procedures. 

These  procedural  implications  are  of  some  importance  with  regard  to  another  issue:  what 
rating  scales  should  be  used  to  assess  occupational  skill  requirements?  In  the  present  study, 
occupational  skill  requirements  were  assessed  using:  (1)  a level  scale,  (2)  an  importance  scale, 
and  (3)  a job  entry  requirements  scale.  The  evidence  presented  in  the  scale  correlations  indicated 
that  the  level  and  importance  scales  provided  effectively  identical  information  in  describing 
occupational  skill  requirements.  Thus,  if  an  occupation  required  a high  level  of  a skill,  this  skill 
was  also  viewed  as  important  to  performance  on  the  job. 

This  finding  suggests  that  either  the  level  or  the  importance  scale  could  be  dropped  in 
future  studies.  Because  importance  ratings  were  collected  in  the  context  of  level  ratings,  the 
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reliability  and  meaningfulness  of  importance  ratings  may  be  contingent  on  the  level  ratings 
collected  earlier.  Further,  the  behavioral  anchors  attached  to  the  level  scale  serve  to  clarify  and 
extend  initial  definitions  of  the  skill.  Based  on  these  considerations,  it  appears  more  appropriate 
to  retain  the  level  scale,  if  a scale  must  be  dropped  in  future  studies. 

The  level  scale  was  not  as  strongly  related  to  the  job  entry  requirements  scale  as  it  was  to 
the  importance  scale.  Thus,  it  is  possible  that  the  job  entry  requirements  scale  might  provide 
some  unique  descriptive  information.  The  job  entry  requirements  scale,  however,  was  less 
reliable  than  the  level  scale,  indicating  that  the  level  scale  provides  a better  single  measure  of 
occupational  skill  requirements.  If  the  job  entry  requirements  scale  is  retained  in  future  studies,  it 
should,  therefore,  be  treated  as  a supplement  to  level  ratings.  Because  this  is  a relatively  simple 
rating  requiring  very  little  time,  we  think  it  should  be  retained. 

Not  only  do  our  findings  have  some  bearing  on  the  rating  scales  that  should  be  used  in 
future  research,  they  also  have  some  noteworthy  implications  for  the  scoring  of  skill  ratings.  In 
the  present  effort,  a number  of  analyses  were  conducted  examining  the  effects  of  scoring  not 
relevant  responses  and  whether  not  relevant  responses  provided  viable  descriptive  information. 
Broadly  speaking,  the  findings  obtained  in  these  analyses  indicate  that  the  inclusion  of  not 
relevant  responses  produced  more  reliable  ratings  that  provided  sensible  patterns  of  results. 
Moreover,  the  not  relevant  responses  provided  some  unique  descriptive  information  Thus,  it 
appears  that  not  relevant  responses  should  be  collected  and  scored  in  further  studies  using  the 
level  scales  to  describe  occupational  skill  requirements. 

Having  addressed  the  rating  scales  that  should  be  used,  and  how  these  scales  should  be 
scored,  in  describing  occupational  skill  requirements,  a new  issue  comes  to  the  fore.  Specifically, 
who  should  use  these  rating  scales?  Completing  the  questionnaire,  as  it  is  cuirently  administered. 
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by  mail  in  paper-and-pencil  format,  does  require  some  degree  of  literacy.  Thus,  these  scales 
should  not  be  used  as  they  stand  by  incumbents  or  analysts  without  some  level  of  reading 
comprehension  skill. 

Although  it  appears  that  most  incumbents  can  make  viable  skill  ratings  using  these  scales, 
in  cases  where  incumbent  data  cannot  be  obtained,  our  findings  suggest  that  analysts  might  be 
used  to  assess  occupational  skill  requirements.  In  this  regard,  however,  a word  of  caution  is  in 
order.  Clearly,  incumbent  and  analysts’  ratings  of  skill  requirements  displayed  good  convergence 
with  regard  to  the  pattern  of  skills  called  for.  Nonetheless,  the  comparative  nature  of  analysts’ 
ratings  resulted  in  lower  mean  scores  than  incumbents'  ratings,  while  incumbents  seemed  to  feel 
that  higher  levels  of  salient  skills  were  called  for  on  their  jobs.  This  pattern  of  findings  suggests 
that  some  control  for  mean  differences  should  be  considered  if  analysts’  ratings  are  to  be  used  in 
lieu  of  incumbents’  ratings.  Further,  given  the  existence  of  these  mean  differences,  incumbents’ 
ratings  should  be  preferred  whenever  our  concern  is  describing  the  occupation  at  hand  without 
reference  to  other  occupations. 

These  observations  about  operational  uses  of  the  scales  bring  us  to  a final  concern  likely 
to  arise  in  routine  application  of  the  level  scale.  Specifically,  would  it  be  possible  to  obtain 
similar  descriptive  information  using  a smaller  set  of  skills?  The  correlations  observed  among  the 
skill  ratings  would  seem,  at  least  at  first  glance,  to  justify  some  reduction  in  the  number  of  skills. 
However,  the  results  obtained  in  the  discriminant  analysis  indicate  that  all  of  the  skills 
contributed  to  distinguishing  occupations,  even  in  a relatively  small  sample.  While  it  is 
recognized  that  discriminating  occupations  is  our  ultimate  concern  in  assessing  occupational  skill 
requirements,  these  discriminant  analysis  findings  indicate  that  reductions  in  the  number  of  skills 
being  evaluated  cannot  be  justified  at  present. 
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Substantive  Conclusions 

Another  reason  for  retaining  the  present  set  of  skills  is  that  these  skills  have  a strong 
foundation  in  prior  research  and  reflect  a cohesive  set  of  attributes  required  for  performance  in 
different  domains  (Mumford  & Peterson,  1995).  In  fact,  the  results  obtained  in  the  present  study 
provide  some  compelling  support  for  the  taxonomy  of  basic  and  cross-functional  skills  proposed 
by  Mumford  and  Peterson  (1995).  To  begin,  all  of  these  skills  were  seen  by  both  incumbents  and 
analysts  as  being  required  across  a range  of  occupations.  In  fact,  no  skill  included  in  this 
taxonomy  had  mean  level  ratings,  with  the  possible  exception  of  Programming  (Descriptor  #29), 
which  would  indicate  that  they  were  not  applicable  across  a range  of  occupations. 

Along  similar  lines,  the  descriptive  statistics  indicated  that  the  basic,  problem  solving, 
and  social  skills  were  required  at  a relatively  high  level  across  all  occupations.  This  finding,  of 
course,  provides  some  support  for  the  notion  that  both  learning  and  performance  skills  must  be 
considered  in  any  viable  taxonomy  of  occupational  skill  requirements.  As  an  aside,  however,  this 
finding  also  provides  some  support  for  current  educational  initiatives  which  assert  that  problem 

solving  and  social  skills  must  be  developed,  along  with  basic  skills,  to  prepare  students  for  entry 
into  the  workplace. 

Perhaps  somewhat  more  compelling  evidence  for  this  taxonomy  of  skills  was  provided  by 
the  results  obtained  in  the  principal  components  and  discriminant  analyses.  In  the  principal 
components  analysis,  three  factors,  labeled  cognitive  skills,  technical  skills,  and  organizational 
skills,  were  identified.  These  factors  are  related  to,  but  not  isomorphic  with,  the  traditional 
distinction  between  data,  people,  and  things,  made  in  describing  occupational  skill  requirements. 
This  structure,  moreover,  was  used  by  Mumford  and  Peterson  (1995)  as  a basis  for  identifying 
the  performance  domains  to  be  covered  by  the  skill  scales,  and  the  confirmation  of  this 
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assumption  in  our  factor  analysis  findings  provides  some  crucial  empirical  evidence  justifying 
both  this  assumption  ana  the  procedures  used  in  developing  the  skills  taxonomy. 

The  discriminant  analysis  is  of  interest  because  it  extends  the  factor  analytic  findings  by 
showing  how  skills  operate  together  to  distinguish  among  occupations.  The  findings  obtained  in 
the  discriminant  analysis  indicate  that  skills  distinguish  among  occupations  based  on  patterns  of 
skills  needed  to  perform  certain  core  duties.  Thus,  functions  were  identified,  such  as  technical 
repair,  research  and  development,  machine  production,  and  system  direction,  which  involved 
multiple  sets  of  interrelated  skills.  This  finding  is  of  some  importance  because  it  indicates  that  an 
occupation  cannot  be  described  with  reference  to  one  or  two  skills.  Instead,  it  suggests  that 
occupations  are  differentiated  in  terms  of  patterns  of  skill  requirements. 

Based  on  the  findings  obtained  in  this  initial  study,  it  appears  that  the  taxonomy  of  basic 
and  cross-functional  skills  proposed  by  Mumford  and  Peterson  (1995)  provides  a reasonably 
comprehensive  system  for  identifying  the  patterns  of  skill  requirements  that  distinguish 
occupations.  For  example,  in  the  discriminant  analyses  it  was  found  that  the  vast  majority  of 
individuals  could  be  accurately  allocated  to  their  occupations  based  on  the  pattern  of  their  level 
ratings  for  the  basic  and  cross-functional  skills.  Given  the  systematic  procedures  used  in 
developing  this  taxonomy,  this  finding,  while  gratifying,  is  hardly  surprising.  By  the  same  token, 
however,  it  should  be  recognized  that  changes  in  job  functions  or  the  need  to  consider  certain 
unique  functions  may  require  the  addition  of  some  new  skills  to  this  initial  taxonomic  system. 

In  considering  the  comprehensiveness  of  this  taxonomy  of  basic  and  cross-functional 
skills,  it  is  important  to  bear  in  mind  an  assumption  underlying  the  development  of  this 
taxonomy.  Specifically,  these  taxonomies  were  expressly  developed  to  capture  those  skills 
involved  in  broad  performance  domains  which  are  likely  to  occur  on  most  jobs.  Thus,  some 
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caution  is  called  for  in  extending  this  taxonomy  to  more  narrowly  defined  domains  or  specific 
occupations,  although,  as  Mumford  (1995)  points  out,  in  many  cases,  these  occupation-specific 
skills  may  reflect  instantiations  of  the  basic  and  cross-functional  skills  within  the  context  of  a 
particular  job  or  set  of  job  tasks.  Bearing  this  point  in  mind,  however,  it  appears  that  the 
proposed  taxonomy  of  basic  and  cross-functional  skills  provides  an  adequate  basis  for  assessing 
the  similarities  and  differences  among  occupations  with  respect  to  general  basic  and  cross- 
functional skill  requirements. 
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Figure  3-1 

Descriptions  and  Definitions  of  Basic  and  Cross-Functional  Skills 
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Figure  3-1  (continued) 

Descriptions  and  Definitions  of  Basic  and  Cross-Functional  Skills 
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Figure  3-1  (continued) 

Descriptions  and  Definitions  of  Basic  and  Cross-Functional  Skills 
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Figure  3-1  (continued) 

Descriptions  and  Definitions  of  Basic  and  Cross-Functional  Skills 
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Figure  3-1  (continued) 

Descriptions  and  Definitions  of  Basic  and  Cross-Functional  Skills 
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Figure  3-1  (continued) 

Descriptions  and  Definitions  of  Basic  and  Cross-Functional  Skills 
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Figure  3-2 

Example  Page  From  the  Skills  Questionnaire 


1.  Reading 

Comprehension 


Understanding  written  sentences  and  paragraphs  in 
work  related  documents. 


Level 

What  level  of  this  skill  is  needed  to  perform  this  job? 


HIGH 


LOW 


©- 

©" 

0-  « 

0-  * 

•©■L 

Not  relevant  at  all  for  performance  on  this  job 


Reading  a scientific  journal  article 
describing  surgical  procedures. 


Reading  a memo  from  management 
describing  new  personnel  policies. 


Reading  step-by-step  instructions  for 
completing  a form. 


ImDortance 

How  important  is  this  skill  to  performance  on  this  job? 

Not 

Important 

Somewhat  Very 

Important  Important  Important 

Extremely 

Important 

© 

© © © 

© 

JobJEntn-  Requirement 
Is  this  level  of  skill  required  for  entry  to  this  job? 


©YES,  it  is  required  for  entry  on  the  job.  (5^  NO,  it  can  be  learned  on  the  job. 


Figure  3-3 

Thirty-Five  Occupations  With  Four  or  More  Incumbents  Completing  the  Skills  Questionnaire 


Occupation 

Code 

Occupation  Title 

Number  of 
Respondents 

15005 

Education  Administrators 

9 

19005 

General  Managers  & Top  Executives 

45 

22135 

Mechanical  Engineers 

6 

25105 

Computer  Programmers 

5 

27311 

Recreation  Workers 

5 

31303 

Teachers,  Preschool 

4 

31305 

Teachers,  Elementary  School 

13 

31502 

Librarians,  Professional 

4 

32502 

Registered  Nurses 

32 

32902 

Medical  & Clinical  Laboratory  Technologists 

4 

49008 

Salespersons,  Except  Scientific  & Retail 

11 

49011 

Salespersons,  Retail 

13 

49017 

Counter  & Rental  Clerks 

4 

49021 

Stock  Clerks,  Sales  Floor 

8 

49023 

Cashiers 

28 

51002 

First  Line  Supervisors,  Clerical/Admiilistrative 

51 

53121 

Loan  & Credit  Clerks 

4 

I 53311 

Insurance  Claims  Clerks 

7 

53905 

Teachers’  Aides  & Assistants,  Clerical 

8 

• 55108 

Secretaries,  Except  Legal  & Medical 

90 

55305 

Receptionists  & Information  Clerks 

12 

55338 

Bookkeeping,  Accounting,  & Auditing  Clerks 

32 

55347 

General  Office  Clerks 

73 

61005 

Police  & Detective  Supervisors 

13 

63014 

Police  Patrol  Officers 

21 

65008 

Waiters  & Waitresses 

16 

65038 

Food  Preparation  Workers 

17 

66008 

Nursing  Aides,  Orderlies,  & Attendants 

18 

67005 

Janitors  & Cleaners 

23 

85132 

Maintenance  Repairers,  General  Utility 

27 

87902 

Earth  Drillers,  Except  Oil  & Gas 

7 

89108 

Machinists 

4 

92974 

Packaging  & Filling  Machine  Operators 

8 

97102 

Truck  Drivers,  Heavy  or  Tractor  Trailer 

17 

97111 

Bus  Drivers,  Schools 

9 

Table  3-1 

Descriptive  Statistics  Across  All  Occupations  and  Reliability  Estimates  for  Rated  Differences 
Between  Occupations:  Basic  and  Cross-Functional  Skills 

Variable 


Job  Entry 

Level Importance Requirement 


Descriptor 

M 

SD  SEM 

a b 

Ik 

M 

SD 

SEM  n- 

M 

SD  SEM  n- 

1.  Reading  Comprehension  4.42 

0.89 

.36 

.84 

3.65 

0.61 

.25 

.83 

1.18 

0.20  .11 

.71 

2.  Active  Listening 

4.28 

0.95 

.39 

.83 

3.63 

0.56 

.26 

.78 

1.34 

0.21  -.12 

.66 

3.  Writing 

3.82 

1.07 

.34 

.90 

3.22 

0.64 

.23 

.87 

1.36 

0.27  .11 

.82 

4.  Speaking 

4.20 

0.80 

.38 

.77 

3.54 

0.49 

.24 

.75 

1.33 

0.22  .13 

.65 

5.  Mathematics 

3.26 

1.16 

.50 

.82 

2.96 

0.65 

.33 

.74 

1.37 

0.25  .14 

.67 

6.  Science 

1.53 

1.35 

.42 

.90 

1.88 

0.82 

.26 

.90 

1.78 

0.27  .09 

.88 

7.  Critical  Thinking 

3.68 

1.02 

.46 

.80 

3.12 

0.70 

.30 

.81 

1.50 

0.21  .11 

.72 

8.  Active  Learning 

3.62 

1.14 

.49 

.81 

3.01 

0.67 

.31 

.79 

1.59 

0.22  ,12 

.72 

9.  Learning  Strategies 

3.83 

0.91 

.42 

.78 

3.12 

0.50 

21 

.71 

1.62 

0.22  .12 

.70 

10.  Monitoring 

3.26 

1.19 

.42 

.88 

2.81 

0.65 

21 

.83 

1.66 

0.22  .12 

.71 

11.  Social  Perceptiveness 

3.62 

1.08 

.44 

.83 

3.06 

0.61 

.30 

.76 

1.62 

0.19  .13 

.52 

12.  Coordination 

3.57 

1.07 

.48 

.80 

2.99 

0.63 

.31 

.75 

1.66 

0.19  .12 

.59 

13.  Persuasion 

3.19 

1.15 

.42 

.86 

2.65 

0.68 

.26 

.85 

1.75 

0.21  .11 

.72 

14.  Negotiation 

2.90 

1.26 

.44 

.88 

2.62 

0.69 

.28 

.84 

1.75 

0.20  .10 

.75 

15.  Instructing 

3.78 

1.08 

.49 

.79 

3.18 

0.57 

.30 

.71 

1.65 

0.21  .11 

.73 

16.  Service  Orientation 

3.28 

1.02 

.51 

.75 

2.86 

0.67 

.35 

.73 

1.69 

0.18  .11 

.61 

17.  Problem  Identification 

4.16 

1.08 

.43 

.84 

3.45 

0.62 

.29 

.79 

1.59 

0.19  .11 

.66 

18.  Information  Gathering 

19.  Information 

3.92 

1.07 

.43 

.84 

3.25 

0.72 

.30 

.83 

1.64 

0.21  .13 

.62 

Organization 
20.  Synthesis/ 

3.21 

1.11 

.51 

.78 

2.79 

0.65 

.32 

.76 

1.69 

0.21  .14 

.54 

Reorganization 

3.24 

1.15 

.45 

.84 

2.79 

0.60 

.28 

.78 

1.70 

0.20  .11 

.66 

21.  Idea  Generation 

3.33 

1.26 

.45 

.87 

2.79 

0.69 

.26 

.86 

1.72 

0.23  .10 

.80 

22.  Idea  Evaluation 

3.07 

1.25 

.46 

.86 

2.67 

0.72 

.30 

.83 

1.73 

0.25  .11 

.80 

23.  Implementation  Planning  2.83 

1.39 

.47 

.88 

2.54 

0.84 

.29 

.88 

1.77 

0.22  .10 

.80 

24.  Solution  Appraisal 

3.02 

1.40 

.44 

.90 

2.68 

0.76 

.29 

.86 

1.75 

0.22  .11 

.77 

25.  Operations  Analysis 

2.02 

1.37 

.56 

.83 

2.06 

0.79 

.33 

.82 

1.84 

0.16  .09 

.65 

26.  Technology  Design 

1.76 

1.40 

.51 

.87 

1.92 

0.75 

.30 

.84 

1.84 

0.17  .10 

.63 

27.  Equipment  Selection 

2.57 

1.32 

.57 

.82 

2.36 

0.77 

.33 

.81 

1.82 

0.14  .10 

.50 

28.  Installation 

1.28 

1.54 

.46 

.91 

1.69 

0.88 

.24 

.92 

1.90 

0.15  .06 

.83 

29.  Programming 

0.83 

1.20 

.57 

.78 

1.49 

0.78 

.33 

.82 

1.90 

0.17  .10 

.65 

30.  Testing 

1.53 

1.60 

.60 

.86 

1.85 

0.95 

.34 

.87 

1.86 

0.17  .09 

.70 

31.  Operation  Monitoring 

1.69 

1.75 

.50 

.92 

1.97 

0.94 

.29 

.90 

1.86 

0.16  .09 

.73 

32.  Operation  and  Control 

2.05 

1.39 

.56 

.84 

2.13 

0.80 

.35 

.80 

1.84 

0.16  .09 

.71 

33.  Product  Inspection 

2.31 

1.34 

.62 

.78 

2.34 

0.75 

.36 

.77 

1.80 

0.16  .09 

.68 

Table  3-1  (continued) 

Descriptive  Statistics  Across  All  Occupations  and  Reliability  Estimates  for  Rated  Differences 
Between  Occupations:  Basic  and  Cross-Functional  Skills 

Variable 

Job  Entry 

Level Importance Requirement 

Descriptor M SD  SEMa  £kb  M SD  SEM  & M SD  SEM  r„ 

34.  Equipment 


Maintenance 

1.67 

1.63 

.47 

.92 

1.95 

0.96 

.26 

.93 

1.88 

0.17 

.07 

.84 

• 35.  Troubleshooting 

2.25 

1.74 

.61 

.88 

2.27 

1.01 

.36 

.88 

1.83 

0.16 

.08 

.74 

36.  Repairing 

1.33 

1.49 

.42 

.92 

1.75 

0.85 

.25 

.91 

1.89 

0.15 

.06 

.82 

37.  Visioning 

2.34 

1.15 

.48 

.82 

2.31 

0.63 

.30 

.78 

1.80 

0.19 

.10 

.76 

38.  Systems  Perception 

2.22 

1.23 

.48 

.84 

2.27 

0.68 

.31 

.79 

1.80 

0.20 

.10 

.77 

39.  Identification  of 
Downstream 

Consequences 

2.19 

1.13 

.44 

.85 

2.22 

0.65 

.26 

.84 

1.81 

0.18 

.09 

.75 

40.  Identification  of  Key 

Causes 

3.17 

1.17 

.43 

.86 

2.74 

0.66 

.27 

.84 

1.77 

0.19 

.10 

.75 

41.  Judgment  and  Decision 

Making 

2.93 

1.33 

.44 

.89 

2.70 

0.79 

.29 

.86 

1.70 

0.24 

.10 

.84 

42.  Systems  Evaluation 

2.02 

1.36 

.46 

.89 

2.09 

0.75 

.28 

.87 

1.84 

0.15 

.08 

.73 

43.  Time  Management 

3.61 

1.18 

.48 

.83 

3.20 

0.67 

.29 

.81 

1.64 

0.24 

.12 

.76 

44.  Management  of 

Financial  Resources 

1.84 

1.50 

.50 

.89 

2.10 

0.90 

.32 

.87 

1.81 

0.22 

.10 

.80 

45.  Management  of 

Material  Resources 

1.94 

1.29 

.47 

.87 

2.09 

0.73 

.28 

.85 

1.85 

0.18 

.08 

.77 

46.  Management  of 

Personnel  Resources 

2.67 

1.38 

.45 

.89 

2.57 

0.80 

.30 

.86 

1.75 

0.23 

.09 

.86 

Note.  Statistics  are  based  on  35  occupations  with  Skills  questionnaire  responses  from  at  least  4 
incumbents  (mean  number  of  incumbents  = 18.51,  median  = 12,  harmonic  mean  = 9.01). 

^his  estimate  of  the  standard  error  of  measurement  was  calculated  as  SEM  = SD*  ^(1-rv) . 

^is  estimate  of  reliability  was  obtained  by  calculating  the  intraclass  correlation  for  k ratings 
across  occupations:  |CC(l,k)  = fBMS-WMSI/BMS  (Shrout  & Fleiss,  1979),  where  k is  the 
harmonic  mean  of  the  number  of  ratings  provided  on  each  job. 


Table  3-2 


O 

ERIC 


Basic  and  Cross-Functional  Skills 
Descriptor 

Level 

_ a b 

II  130 

Variable 

Importance 

II  130 

Job  Entry 
Reauirement 

II  130 

1. 

Reading  Comprehension 

37 

95 

36 

94 

21 

89 

2. 

Active  Listening 

36 

94 

29 

92 

18 

87 

3. 

Writing 

49 

97 

42 

96 

34 

94 

4. 

Speaking 

27 

92 

25 

91 

17 

86 

5. 

Mathematics 

33 

94 

24 

91 

18 

87 

6. 

Science 

51 

97 

50 

97 

44 

96 

7. 

Critical  Thinking 

30 

93 

32 

94 

22 

89 

8. 

Active  Learning 

33 

94 

29 

92 

22 

90 

9. 

Learning  Strategies 

28 

92 

21 

89 

21 

89 

10. 

Monitoring 

44 

96 

35 

94 

21 

89 

11. 

Social  Perceptiveness 

36 

94 

26 

91 

11 

78 

12. 

Coordination 

30 

93 

25 

91 

14 

83 

13. 

Persuasion 

41 

96 

39 

95 

22 

89 

14. 

Negotiation 

44 

96 

37 

95 

25 

91 

15. 

instructing 

30 

93 

22 

89 

23 

90 

16. 

Service  Orientation 

25 

91 

23 

90 

15 

84 

17. 

Problem  Identification 

37 

95 

29 

93 

18 

87 

18. 

Information  Gathering 

37 

95 

35 

94 

16 

85 

19. 

Information  Organization 

29 

92 

26 

91 

12 

80 

20. 

Synthesis/  Reorganization 

37 

95 

28 

92 

18 

86 

21. 

Idea  Generation 

44 

96 

40 

95 

31 

93 

22. 

Idea  Evaluation 

42 

96 

35 

94 

30 

93 

23. 

Implementation  Planning 

46 

96 

45 

96 

31 

93 

24. 

Solution  Appraisal 

50 

97 

40 

95 

27 

92 

25. 

Operations  Analysis 

35 

94 

34 

94' 

17 

86 

26. 

Technology  Design 

42 

96 

38 

95 

16 

85 

27. 

Equipment  Selection 

33 

94 

33 

94 

10 

77 

28. 

Installation 

53 

97 

57 

98 

34 

94 

29. 

Programming 

28 

92 

34 

94 

17 

86 

30. 

Testing 

40 

95 

43 

96 

21 

89 

31. 

Operation  Monitoring 

55 

97 

51 

97 

23 

90 

32. 

Operation  and  Control 

36 

94 

31 

93 

21 

89 

33. 

Product  Inspection 

29 

92 

27 

92 

19 

88 

34. 

Equipment  Maintenance 

55 

97 

58 

98 

36 

94 

35. 

Troubleshooting 

44 

96 

44 

96 

24 

90 

36. 

Repairing 

56 

97 

54 

97 

33 

94 

37. 

Visioning 

34 

94 

28 

92 

26 

91 

38. 

Systems  Perception 

37 

95 

29 

92 

27 

92 

39. 

Identification  of  Downstream  Consequences 

39 

95 

36 

94 

25 

91 

40. 

Identification  of  Key  Causes 

L.  i 

• V 

162 

41 

95 

36 

94 

25 

91 

Table  3-2  (continued) 

Reliability  of  Rated  Differences  Between  Occupations  Considering  Varying  Numbers  nf  Raters: 
Basic  and  Cross-Functional  Skills 


Variable 


Descriptor 


Level 


Job  Entry 

Importance  Requirement 


41. 

Judgment  and  Decision  Making 

47 

96 

41 

95 

LI 

37 

±30 

95 

42. 

Systems  Evaluation 

46 

96 

42 

96 

23 

90 

43. 

Time  Management 

36 

94 

32 

93 

26 

92 

44. 

Management  of  Financial  Resources 

47 

96 

43 

96 

31 

93 

45. 

Management  of  Material  Resources 

42 

96 

39 

95 

27 

92 

46. 

Management  of  Personnel  Resources 

48 

97 

41 

95 

40 

95  ' 

Note.  Reliability  estimates  are  based  on  35  occupations  with  Skills  questionnaire  responses 
from  at  least  4 incumbents  (mean  number  of  incumbents=  18.51,  median=12,  harmonic 
mean=9.01).  Decimals  are  omitted. 

Single  rater  estimates  of  reliability  were  obtained  by  calculating  the  intraclass  correlation  for 
single  judge  ratings  across  occupations:  ICC(U)=rBMS-WMSl/rBMS+(k-l)WMSl  (Shrout  & 
Heiss,  1979),  where  k is  the  harmonic  mean  of  the  number  of  ratings  provided  on  each  job. 
^Estimates  of  reliability  for  30  raters  were  obtained  by  applying  the  Spearman-Brown  correction 
formula  to  the  single  rater  reliability  estimates. 
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Table  3-3 

Reliability  of  Rated  Differences  Between  Occupations  Considering  Various  Recoding  Schemes- 
Basic  and  Cross-Functional  Skills 

Type  of  Scale  and  Recoding  Scheme  Applied 


Job  Entry 

Level  Importance  Requirement 


Descriptor 

L 

lb 

Ec 

la 

lb 

Ea 

Eb 

1. 

Reading  Comprehension 

84 

80 

50 

83 

80 

71 

64 

2. 

Active  Listening 

83 

77 

69 

78 

61 

66 

50 

3. 

Writing 

90 

85 

73 

87 

79 

82 

72 

4. 

Speaking 

77 

75 

53 

75 

69 

65 

55 

5. 

Mathematics 

82 

74 

70 

74 

60 

67 

43 

6. 

Science 

90 

60 

87 

90 

54 

88 

46 

7. 

Critical  Thinking 

80 

71 

62 

81 

70 

72 

55 

8. 

Active  Learning 

81 

74 

64 

79 

63 

72 

56 

9. 

Learning  Strategies 

78 

79 

54 

71 

67 

70 

63 

10. 

Monitoring 

88 

84 

81 

83 

66 

71 

59 

11. 

Social  Perceptiveness 

83 

84 

61 

76 

70 

52 

48 

12. 

Coordination 

80 

72 

64 

75 

52 

59 

41 

13. 

Persuasion 

86 

77 

73 

85 

75 

72 

60 

14. 

Negotiation 

88 

82 

68 

84 

73 

75 

60 

15. 

Instructing 

79 

78 

58 

71 

69 

73 

70 

16. 

Service  Orientation 

75 

68 

66 

73 

52 

61 

48 

17. 

Problem  Identification 

84 

80 

62 

79 

69 

66 

58 

18. 

Information  Gathering 

84 

77 

69 

83 

73 

62 

.53 

19. 

Information  Organization 

78 

63 

72 

76 

53 

54 

47 

20. 

Synthesis/  Reorganization 

84 

79 

78 

78 

61 

66 

61 

21. 

Idea  Generation 

87 

84 

70 

86 

78 

80 

70 

22. 

Idea  Evaluation 

86 

84 

74 

83 

71 

80 

72 

23. 

Implementation  Planning 

88 

81 

78 

88 

76 

80 

67 

24. 

Solution  Appraisal 

90 

85 

80 

86 

64 

77 

61 

25. 

Operations  Analysis 

83 

74 

78 

82 

54 

65 

38 

26. 

Technology  Design 

87 

58 

84 

84 

33 

63 

10 

27. 

Equipment  Selection 

82 

53 

78 

81 

53 

50 

06 

28. 

Installation 

91 

61 

87 

92 

64 

83 

40 

29. 

Programming 

78 

42 

63 

82 

49 

65 

08 

30. 

Testing 

86 

41 

83  ' 

87 

63 

70 

25 

31. 

Operation  Monitoring 

92 

76 

84 

90 

69 

73 

13 

32. 

Operation  and  Control 

84 

85 

68 

80 

75 

71 

70 

33. 

Product  Inspection 

78 

77 

71 

77 

56 

68 

51 

34. 

Equipment  Maintenance 

92 

76 

84 

93 

79 

84 

56 

35. 

Troubleshooting 

88 

80 

75 

88 

79 

74 

55 

36. 

Repairing 

92 

77 

83 

91 

70 

82 

41 

37. 

Visioning 

82 

79 

61 

78 

61 

76 

68 

38. 

Systems  Perception 

84 

81 

64 

79 

58 

77 

67 

39. 

Identification  of  Downstream  Consequences  85 

75 

72 

84 

64 

75 

58 

Table  3-3  (continued) 

Reliability  of  Rated  DifferencesBetween  Occupations  Considering  Various  Recodins  Schemes* 
Basic  and  Cross-Functional  Skills 

Type  of  Scale  and  Recoding  Scheme  Applied 


Job  Entry 

Level  Importance  Requirement 


Descriptor 

la 

Sb 

Ic 

S a 

lb 

S a 

Sb 

40. 

Identification  of  Key  Causes 

86 

81 

69 

84 

74 

75 

62 

41. 

Judgment  and  Decision  Making 

89 

82 

75 

86 

76 

84 

72 

42. 

Systems  Evaluation 

89 

79 

77 

87 

69 

73 

41 

43. 

Time  Management 

83 

73 

75 

81 

62 

76 

66 

44. 

Management  of  Financial  Resources 

89 

75 

83 

87 

61 

80 

51  * 

45. 

Management  of  Material  Resources 

87 

76 

84 

85 

55 

77 

54 

46. 

Management  of  Personnel  Resources 

89 

87 

71 

86 

78 

86 

78 

Reliability  estimates  are  based  on  35  occupations  with  Skills  questionnaire  responses  from 
at  least  4 incumbents  (mean  number  of  incumbents=  18.51,  median=12,  harmonic  mean=9.01). 
Reliability  estimates  stipulated  as  r&  were  calculated  using  the  full  eight  point  scale  for  level,  and 
retaining  all  of  the  data  for  the  importance  and  "Job  Entry  Requirement"  scales.  Reliability 
estimates  stipulated  as  were  calculated  using  a reduced  seven  point  scale  for  level,  and 
excluding  the  data  for  the  importance  and  "Job  Entry  Requirement"  scales  where  the  rater 
marked  "NR"  on  the  level  scale.  Reliability  estimates  stipulated  as  rc  were  calculated  using  a 
binary  coded  scale  for  level  (relevant/not  relevant).  Decimals  are  omitted. 
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Table  3-4a 

Analysis  of  Variance  for  Descriptor.  Occupation,  and  Relevant  Interactions  as  Sources  of 
Variation  on  the  Level  Scale:  Basic  and  Cross-Functional  Skills 


Source  of  Variation 

SS 

df 

MS 

F 

Occupations 

20282.98 

34 

596.56 

10.57* 

S (Occupations) 

34594.09 

613 

56.43 

Descriptor 

12023.43 

45 

267.19 

135.00* 

Descriptor  x Occupations 

14750.96 

1530 

9.64 

4.87* 

Descriptor  x S (Occupations) 

54593.81 

27585 

1.98 

Note.  Occupations  are  treated  as  random,  between-subjects  effects,  while  descriptors  are  treated 
as  fixed,  within-subjects  effects. 

*P  < .05 


Table  3-4b 

Analysis  of  Variance  for  Descriptor.  Occupation,  and  Relevant  Interactions  as  Sources  of 
Variation  on  the  Importance  Scale:  Basic  and  Cross-Functional  Skills 


Source  of  Variation 

SS 

df 

MS 

F 

Occupations 

5393.96 

34 

158.65 

8.81* * 

S (Occupations) 

11034.76 

613 

18.00 

Descriptor 

4341.32 

45 

96.47 

121.81* 

Descriptor  x Occupations 

5764.59 

1530 

3.77 

4.76* 

Descriptor  x S(Occupations) 

21846.80 

27585 

0.79 

Note,  Occupations  are  treated  as  random,  between-subjects  effects,  while  descriptors  are  treated 
as  fixed,  within-subjects  effects. 

*2  < .05 


Table  3-4c 

Analysis  of  Variance  for  Descriptor,  Occupation,  and  Relevant  Interactions  as  Sources  of 
Variation  on  the  Job  Entry  Requirement  Scale:  Basic  and  Cross-Functional  Slrills 


Source  of  Variation 

SS 

df 

MS 

F 

Occupations 

507.02 

34 

14.91 

6.70* * 

S(Occupations) 

1364.67 

613 

2.23 

Descriptor 

398.16 

45 

8.85 

78.47* 

Descriptor  x Occupations 

429.84 

1530 

0.28 

2.49* 

Descriptor  x S(Occupations) 

3110.58 

27585 

0.11 

Note.  Occupations  are  treated  as  random,  between-subjects  effects,  while  descriptors  are 
treated  as  fixed,  within-subjects  effects. 

*2  < .05  . 


Table  3-5 

Interrater  Agreement  Coefficients  for  Each  Scale  Type:  Basic  and  Cross-Functional  gkffls 

Number  of  Raters  on  Each  Variable 

Scale  Type  & £l  ^ 

Level  79  30  95- 

Importance  79  29  93 

Job  Entry  Requirement  60  14  83 

NPle-  Interrater  agreement  coefficient  estimates  are  based  on  35  occupations  with  Skills 
questionnaire  responses  from  at  least  4 incumbents  (mean  number  of  incumbents=  18.51, 
median=12,  harmonic  mean=9.01).  Full  sample  interrater  agreement  coefficients  (a)  were 
obtained  by  considering  the  "Descriptor  x Occupations"  tern  from  Tables  3-4a,  3-4b,  and  3-4c  as 
true  variance.  Error  variance  was  defined  as  the  "Descriptor  x S (Occupations)"  term.’  Estimates 
of  reliability  for  1 and  30  raters  were  obtained  by  applying  the  Spearman-Brown  correction 
formula  to  the  k rater  reliability  estimates,  where  k is  the  harmonic  mean  of  the  number  of  raters 
for  each  occupation.  Decimals  are  omitted. 


Table  3-6a 


* 7 JU  WA — T mianvv  xui  -ipiVJl , Vj/CUUpd.UUll,  3JlQ  IvCiCVdill  UlLCJ 

Sources  of  Variation  on  the  Level  Scale:  Basic  and  Cross-Functional  Stills 

-actions  as 
F 

Source  of  Variation 

SS 

df 

MS 

Occupations 

3416.92 

34 

100.50 

11.14* 

S (Occupations) 

5530.07 

613 

9.02 

Aggregate 

934.74 

6 

155.79 

216.00* 

Aggregate  x Occupations 

1016.73 

204 

4.98 

6.91* 

Aggregate  x S (Occupations) 

2652.75 

3678 

0.72 

Note.  Occupations  are  treated  as  random,  between-subjects  effects,  while  aggregate 
descriptors  are  treated  as  fixed,  within-subjects  effects. 

*2  <.05 


Table  3-6b 


OI  vanance  ror  Aggregate  uescnntor,  Occuoation.  and  Relevant  Interactions  as 
Sources  of  Variation  on  the  Importance  Scale:  Basic  and  Cross-Functional  Skills 

Source  of  Variation 

SS 

df 

MS 

F 

Occupations 

922.90 

34 

27.14 

9.28* 

S(Occupations) 

1792.61 

613 

2.92 

Aggregate 

322.10 

6 

53.68 

201.74* 

Aggregate  x Occupations 

382.77 

204 

1.88 

7.05* 

Aggregate  x S(Occupations) 

978.72 

3678 

0.27 

Note.  Occupations  are  treated  as  random,  between-subjects  effects,  while  aggregate 
descriptors  are  treated  as  fixed,  within-subjects  effects. 

*E  < .05 
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Table  3-6c 


^ mui  7 t uwuuuduuii,  diiu.  tvcicvdiii  mieraci 

Sources  of  Variation  on  the  Importance  Scale:  Basic  and  Cross-Functional  Skills 

[ions  as 
F 

Source  of  Variation 

SS 

df 

MS 

Occupations 

92.92 

34 

2.73 

7.44* 

S (Occupations) 

225.23 

613 

0.37 

Aggregate 

43.89 

6 

7.31 

174.08* 

Aggregate  x Occupations 

28.07 

204 

0.14 

3.27 

Aggregate  x S (Occupations) 

154.55 

3678 

0.04 

Note.  Occupations  are  treated  as  random,  between-subjects  effects,  while  aggregate 
descriptors  are  treated  as  fixed,  within-subjects  effects. 

*P  < .05 
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Table  3-7 

Interrater  Agreement  Coefficients  for  Aggregate  Descriptors  for  Each  Scale  Type:  Basic  and 
Cross-Functional  Skills 


Scale  Type 

Number  of  Raters  on  Each  Variable 

£k 

r i 

130 

Level 

86 

40 

95 

Importance 

86 

40 

95 

Job  Entry  Requirement 

69 

20 

88 

Note.  Interrater  agreement  coefficient  estimates  are  based  on  35  occupations  with  Skills 
questionnaire  responses  from  at  least  4 incumbents  (mean  number  of  incumbents=  18.51, 
median=12,  harmonic  mean=9.01).  Full  sample  interrater  agreement  coefficients  (q.)  were 
obtained  by  considering  the  "Aggregate  x Occupations"  term  from  Tables  3-6a,  3-6b,  and  3-6c  as 
true  variance.  Error  variance  was  defined  as  the  "Aggregate  x S (Occupations)"  term.  Estimates 
of  reliability  for  1 and  30  raters  were  obtained  by  applying  the  Spearman-Brown  correction 
formula  to  the  k rater  reliability  estimates,  where  k is  the  harmonic  mean  of  the  number  of  raters 
for  each  occupation.  Decimals  are  omitted. 


Table  3-8 

Means  and  Standard  Deviations  of  Correlations  Between  Level,  Importance,  and  Joh  Entry 
Requirement  Scales  Across  Occupations  and  Descriptors:  Basic  and  Cross-Functional  Skills 


Scale 

Level 

Importance 

Job  Entry 
Requirement 

na 

M 

SD 

n 

M 

SD 

n M SD 

Level 

— 

— 

— 

35 

.96 

.04 

35  -.71  .18 

Importance 

46 

.95 

.04 

— 

— 

— 

35  -.74  .17 

Job  Entry  Requirement 

46 

-.66 

.11 

46 

-.69 

.12 

— — — 

Note.  All  correlations  were  calculated  based  on  the  mean  of  ratings  assigned  by  raters  for  a given- 
occupation,  descriptor,  and  scale.  Level-importance  means  above  the  diagonal  were  calculated  by 
taking  the  level  scale  means  on  a given  occupation  for  all  descriptors,  correlating  them  with 
importance  scale  means,  for  that  occupation,  and  then  averaging  them  with  the  correlations  for 
other  occupations.  Level-importance  means  below  the  diagonal  were  calculated  by  taking  the 
level  scale  means  on  a given  descriptor  for  all  occupations,  correlating  them  with  importance 
scale  means  for  that  descriptor,  and  averaging  them  with  correlations  for  other  descriptors.  Other 
means  in  the  table  were  calculated  in  a similar  manner. 

“Number  of  correlations  averaged,  not  number  of  observations  on  which  correlations  were 
calculated. 
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Table  3- 10a  (continued) 

Intercorrelations  of  Descriptors  for  the  Level  Scale  (Individual-Level  Data):  Basic  and  Cross-Functional  Skills 
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Table  3- 10a  (continued) 

Intercorrelations  of  Descriptors  for  the  Level  Scale  (Individual-Level  Data):  Basic  and  Cross-Functional  Skills 
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Table  3- 10b  (continued) 

Intercorrelations  of  Descriptors  for  the  Importance  Scale  (Individual-Level  Data):  Basic  and  Cross-Functional  Skills 
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Table  3-11 

Principal  Components  Analysis  Pattern  Matrix  for  the  Level  Scale:  Basic  and  Cross-Functional 
Skills 


Factor 


Descriptor 

FI 

F2 

F3 

Communality 

1. 

Reading  Comprehension 

.82 

.12 

.15 

.71 

2. 

Active  Listening 

.87 

-.17 

.16 

.82 

3. 

Writing 

.89 

-.04 

.24 

.84 

4. 

Speaking 

.85 

.01 

.16 

.75 

5. 

Mathematics 

.56 

.54 

.04 

.60 

6. 

Science 

.58 

.70 

.05 

.82 

7. 

Critical  Thinking 

.85 

.25 

.23 

.84 

8. 

Active  Learning 

.82 

.42 

.19 

.88 

9. 

Learning  Strategies 

.75 

.07 

.28 

.64 

10. 

Monitoring 

.80 

.11 

.40 

.81 

11. 

Social  Perceptiveness 

.67 

-.32 

.39 

.70 

12. 

Coordination 

.80 

.29 

.31 

.82 

13. 

Persuasion 

.80 

.02 

.46 

.86 

14. 

Negotiation 

.58 

-.08 

.68 

.80 

15. 

Instructing 

.74 

.12 

.13 

.59 

16. 

Service  Orientation 

.61 

-.30 

.47 

.68 

17. 

Problem  Identification 

.83 

.22 

.19 

.77 

18. 

Information  Gathering 

.87 

.22 

.22 

.85 

19. 

Information  Organization 

.77 

.28 

.33 

.78 

20. 

Synthesis/Reorganization 

.89 

.16 

.25 

.88 

21. 

Idea  Generation 

.82 

.29 

.28 

.83 

22. 

Idea  Evaluation 

.73 

.36 

.44 

.86 

23. 

Implementation  Planning 

.76 

.18 

.43 

.79 

24. 

Solution  Appraisal 

.75 

.16 

.50 

.83 

25. 

Operations  Analysis 

.60 

.58 

.34 

.82 

26. 

Technology  Design 

.41 

k 

.82 

.23 

.90 

27. 

Equipment  Selection 

.39 

.74 

.33 

.81 

28. 

Installation 

.05 

.90 

.22 

.85 

29. 

Programming 

.23 

.67 

-.07 

.51 

30. 

Testing 

.19 

.94 

.03 

.93 

31. 

Operation  Monitoring 

-.09 

.88 

.15 

.81 

32. 

Operation  and  Control 

-.09 

.87 

.16 

.79 

33. 

Product  Inspection 

.13 

.81 

.30 

-76 

34. 

Equipment  Maintenance 

-.12 

.87 

.17 

.80 

35. 

Troubleshooting 

.15 

.96 

.06 

.95 

36. 

Repairing 

-.02 

.92 

.13 

.86 

37. 

Visioning 

.30 

.50 

.71 

.85 

38. 

Systems  Perception 

.35 

.58 

.66 

.90 

39. 

Identification  of  Downstream  Consequences 

.39 

.34 

.78 

.88 
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Table  3-11  (continued) 

Principal  Components  Analysis  Pattern  Matrix  for  Level  Scale:  Basic  and  Cross-Functional 
Skills 


Descriptor 

FI 

F2 

Factor 

F3 

Communality 

40. 

. Identification  of  Key  Causes 

.66 

.27 

.54 

.80 

41. 

Judgment  and  Decision  Making 

.64 

.29 

.56 

.81 

42. 

Systems  Evaluation 

.45 

.47 

.68 

.89 

43. 

Time  Management 

.74 

.04 

.52 

.82 

44. 

Management  of  Financial  Resources 

.41 

.19 

.81 

.85 

45. 

Management  of  Material  Resources 

.35 

.36 

.80 

.90 

46. 

Management  of  Personnel  Resources 

.42 

.17 

.83 

.90 

Percent  of  Variance 

38 

25 

17 

Eigenvalue 

17.60 

11.56 

7.98 

Note.  N = 35.  The  correlation  matrix  was  based  on  means  calculated  at  the  occupation  level.  FI  = 
Cognitive  Skills,  F2  = Technical  Skills,  F3  = Organizational  Skills.  These  loadings  are  based  on 
an  orthogonal  varimax  rotation. 


Table  3-12a 

Descriptor  Means  and  Standard  Deviations  on  the  Level  Scale  on  Six  Example  Occupations:  Basic  and  Cross-Functional  Skills 
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Table  3- 12a  (continued) 

Descriptor  Means  and  Standard  Deviations  on  the  Level  Scale  on  Six  Example  Occupations:  Basic  and  Cross-Functional  Skills 
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Table  3-12a  (continued) 

Descriptor  Means  and  Standard  Deviations  on  the  Level  Scale  on  Six  Example  Occupations:  Basic  and  Cross-Functional  Skills 
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Table  3- 12b 

Descriptor  Means  and  Standard  Deviations  on  the  Importance  Scale  on  Six  Example  Jobs:  Basic  and  Cross-Functional  Skills 
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Descriptor  Means  and  Standard  Deviations  on  the  Importance  Scale  on  Six  Example  Occupations:  Basic  and  Cross-Functional  Skills 
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Table  3- 12b  (continued) 

Descriptor  Means  and  Standard  Deviations  on  the  Importance  Scale  on  Six  Example  Occupations:  Basic  and  Cross^uncUonal  Skill. 

. Occupations 
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Table  3- 12c 

Descriptor  Means  and  Standard  Deviations  on  the  Job  Entry  Requirements  Scale  on  Six  Example  Occupations:  Basic  and  Cross- 
Functional  Skills 

Occupations  
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• Table  3- 12c  (continued) 

Descriptor  Means  and  Standard  Deviations  on  the  Job  Entry  Requirements  Scale  on  Six  Example  Occupations:  Basic  and  Cross- 
Functional  Skills 

Occupations 
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Table  3-13 

Rotated  Correlations  Between  Discriminating  Variables  and  Canonical  Discriminant  Functions 
for  the  Level  Scale:  Basic  and  Cross-Functional  Skills 

Functions 


Descriptor 

FI 

F2 

F3 

F4 

F5 

F6 

IT2 

2 

T| 

1. 

Reading  Comprehension 

-.02 

.07 

.04 

.05 

.30 

-.01 

.10 

.26 

2. 

Active  Listening 

-.04 

.04 

.03 

.04 

.61 

.08 

.38 

.25 

3. 

Writing 

.06 

.15 

.05 

-.05 

.61 

.12 

.42 

.35 

4. 

Speaking 

.08 

.04 

.00 

.16 

.60 

-.03 

.40 

.20 

5. 

Mathematics 

.07 

.15 

.11 

.07 

.39 

.08 

.20 

.23 

6. 

Science 

.10 

.16 

.14 

.70 

.33 

.12 

.66 

.37 

7. 

Critical  Thinking 

.02 

.11 

.04 

.10 

.61 

.08 

.40 

.21 

8. 

Active  Learning 

.02 

.20 

.04 

.17 

.51 

.08 

.34 

.23 

9. 

Learning  Strategies 

.09 

.09 

.03 

.08 

.28 

.04 

.10 

.20 

10. 

Monitoring 

-.05 

.23 

.07 

.01 

.49 

.02 

.30 

.31 

11. 

Social  Perceptiveness 

-.03 

.11 

.08 

.00 

.51 

-.02 

.28 

.25 

12. 

Coordination 

.12 

.09 

.03 

.02 

.62 

.04 

.41 

.21 

13. 

Persuasion 

.10 

.17 

.07 

.07 

.68 

.05 

.51 

.29 

14. 

Negotiation 

.09 

.26 

.05 

.06 

.57 

.08 

.42 

.31 

15. 

Instructing 

.12 

.09 

.06 

.15 

.37 

.04 

.18 

.21 

16. 

Service  Orientation 

.04 

.13 

.03 

.04 

.37 

.04 

.16 

.18 

17. 

Problem  Identification 

.07 

.09 

.02 

.07 

.51 

-.09 

.29 

.26 

18. 

Information  Gathering 

.10 

.17 

-.04 

.06 

.41 

.07 

.22 

.26 

19. 

Information  Organization 

.07 

.20 

-.02 

.05 

.35 

.25 

.23 

.20 

20. 

Synthesis/Reorganization 

.08 

.27 

-.03 

.02 

.37 

.11 

.23 

.26 

21. 

Idea  Generation 

.05 

.23 

.14 

.02 

.50 

.17 

.36 

.31 

22. 

Idea  Evaluation 

.05 

.35 

.00 

.09 

.36 

.12 

.28 

.29 

23. 

Implementation  Planning 

.05 

.41 

.01 

.05 

.44 

.29 

.45 

.32 

24. 

Solution  Appraisal 

.09 

.35 

.05 

.08 

.37 

.13 

.30 

.36 

25. 

Operations  Analysis 

.09 

.46 

.14 

.09 

.21 

.31 

.38 

.25 

26. 

Technology  Design 

.26 

.35 

.08 

.30 

.12 

.31 

.40 

.29 

27. 

Equipment  Selection 

.18 

.21 

.19 

.15 

.12 

.11 

.16 

.23 

28. 

Installation 

.64 

.29 

.21 

.05 

.07 

.30 

.63 

.38 

29. 

Programming 

.18 

.20 

.10 

.04 

.09 

.74 

.63 

.20 

30. 

Testing 

.25 

.26 

.24 

.20 

.06 

.41 

.40 

.28 

31. 

Operation  Monitoring 

.17 

.10 

.79 

.11 

.09 

.06 

.68 

.40 

32. 

Operation  and  Control 

.11 

.14 

.62 

-.05 

.02 

.11 

.43 

.25 

33. 

Product  Inspection 

.17 

.23 

.31 

.05 

.07 

.04 

.19 

.20 

34. 

Equipment  Maintenance 

.35 

.14 

.36 

.12 

.10 

.05 

.31 

.40 

35. 

Troubleshooting 

.38 

.21 

.37 

.25 

.05 

.22 

.44 

.31 

36. 

Repairing 

.78 

.20 

.21 

.04 

.10 

.15 

.72 

.41 

37. 

Visioning 

.15 

.58 

.00 

-.01 

.11 

.16 

.40 

.24 

38. 

Systems  Perception 

.16 

.60 

.11 

.10 

.06 

.09 

.42 

.26 

39. 

Identification  of 
Downstream  Consequences  .08 

.56 

.12 

.20 

.09 

.09 

.39 

.27 
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Table  3-13  (continued) 

Rotated  Correlations  Between  Discriminating  Variables  and  Canonical  Discriminant  Functions 
for  the  Level  Scale:  Basic  and  Cross-Functional  Skills 


Functions 


Descriptor 

FI 

F2 

F3 

F4 

F5 

F6 

IT2 

„2 

T| 

40. 

Identification  of  Key 

Causes 

.02 

.43 

.05 

.10 

.26 

.00 

.27 

.29 

41. 

Judgment  and  Decision 

Making 

.05 

.42 

.04 

.08 

.23 

.05 

.24 

.33 

42. 

Systems  Evaluation 

.06 

.60 

.08 

.08 

.07 

.15 

.40 

.33 

43. 

Time  Management 

.07 

.29 

.06 

.01 

.30 

-.03 

.18 

.25 

44. 

Management  of  Financial 

Resources 

.05 

.65 

.13 

-.13 

.29 

.08 

.55 

.34 

45. 

Management  of  Material 

Resources 

.14 

.61 

.14 

.10 

.18 

.01 

.45 

.29 

46. 

Management  of  Personnel 

Resources 

.09 

.51 

.06 

-.04 

.28 

.05 

.36 

.34 

Sc 

.81 

.73 

.65 

.61 

.60 

.56 

Percent  of  Variance  23 

14 

9 

7 

7 . 

6 

Eigenvalue 

1.92 

1.15 

.73 

.59 

.56 

.47 

Hote.  Statistics  are  based  on  35  occupations  with  Skills  questionnaire  responses  from  at  least  4 
incumbents  (mean  number  of  incumbents  = 18.51,  median  = 12,  harmonic  mean  = 9.01).  FI  = 
Technical  Repair;  F2  = Systems  Direction;  F3  = Machine  Production;  F4  = Research  and 
Development;  F5  = Interactive  Problem  Solving;  F6  = Computer  Applications. 

T£  = Sum  of  squared  rotated  standardized  discriminant  function  coefficients  across  6 functions, 
•q  *•  Variance  in  Skill  Level  Scale  ratings  accounted  for  by  occupations. 

The  statistics  Rc  “Percent  of  Variance,”  and  “Eigenvalue,”  were  calculated  based  on  the 
unrotated  discriminant  functions. 


O 

ERIC 


2 5$ 


Table  3- 14a 

Comparison  of  Incumbent  and  Analyst  Descriptive  Statistics  Across  AH  Occupations  and 
Reliability  Estimates  for  Rated  Difference  Between  Jobs  for  the  Level  Scale:  Basic  and  Cross- 
Functional  Skills 


Incumbent Analyst 


Descriptor 

M 

SD 

Ik 

M 

SD 

Ik 

t 

F 

lia  d2 

1.  Reading  Comprehension 

4.42 

0.89 

.84 

3.53 

1.15 

.94 

7.77* 

1.66 

.81  1.22 

2.  Active  Listening 

4.28 

0.95 

.83 

3.23 

1.04 

.93 

8.04* 

1.18 

.71  1.66 

3.  Writing 

3.82 

1.07 

.90 

3.11 

1.08 

.94 

5.77* 

1.01 

.77  1.02 

4.  Speaking 

4.20 

0.80 

.77 

3.13 

1.14 

.95 

7.75* 

2.02 

.70  1.79 

5.  Mathematics 

3.26 

1.16 

.82 

2.72 

1.14 

.95 

3.89* 

1.04 

.74  0.95 

6.  Science 

1.53 

1.35 

.90 

1.24 

1.41 

.96 

2.15* 

1.10 

.84  0.67 

7.  Critical  Thinking 

3.68 

1.02 

.80 

2.63 

1.38 

.95 

7.19* 

1.83 

.78  1.84 

8.  Active  Learning 

3.62 

1.14 

.81 

2.49 

1.44 

.96 

7.85* 

1.59 

.81  1.96 

9.  Learning  Strategies 

3.83 

0.91 

.78 

2.38 

1.29 

.94 

9.73* 

2.03 

.73  2.85 

10.  Monitoring 

3.26 

1.19 

.88 

2.78 

1.08 

.93 

4.18* 

1.22 

.83  0.67 

1 1.  Social  Perceptiveness 

3.62 

1.08 

.83 

2.50 

1.30 

.96 

6.82* 

1.46 

.68  2.16 

12.  Coordination 

3.57 

1.07 

.80 

2.56 

1.43 

.95 

5.56* 

1.80 

.67  2.13 

13.  Persuasion 

3.19 

1.15 

.86 

1.64 

1.16 

.93 

10.22* 

1.00 

.70  3.19 

14.  Negotiation 

2.90 

1.26 

:88 

1.46 

1.11 

.93 

9.26* 

1.29 

.71  2.87 

15.  Instructing 

3.78 

1.08 

.79 

1.97 

1.32 

.92 

9.07* 

1.48 

.53  4.61 

16.  Service  Orientation 

3.28 

1.02 

.75 

2.49 

1.09 

.91 

4.63* 

1.14 

.54  1.63 

17.  Problem  Identification 

4.16 

1.08 

.84 

3.15 

1.16 

.94 

7.04* 

1.16 

.72  1.70 

18.  Information  Gathering 

3.92 

1.07 

.84 

2.96 

1.23 

.96 

7.46* 

1.32 

.79  1.49 

19.  Information  Organization 

3.21 

1.11 

.78 

2.95 

1.09 

.93 

1.94 

1.02 

.73  0.70 

20.  Synthesis/  Reorganization 

3.24 

1.15 

.84 

2.13 

1.11 

.93 

8.61* 

1.06 

.77  1.80 

21.  Idea  Generation 

3.33 

1.26 

.87 

2.24 

1.23 

.95 

7.60* 

1.04 

.77  1.87 

22.  Idea  Evaluation 

3.07 

1.25 

.86 

2.31 

1.25 

.96 

5.70* 

1.00 

.80  1.19 

23.  Implementation  Planning 

2.83 

1.39 

.88 

2.04 

1.35 

.94 

4.86* 

1.06 

.75  1.55 

24.  Solution  Appraisal 

3.02 

1.40 

.90 

2.64 

1.17 

.95 

2.86* 

1.43 

.82  0.76 

25.  Operations  Analysis 

2.02 

1.37 

.83 

1.63 

1.27 

.92 

2.62* 

1.16 

.78  0.90 

26.  Technology  Design 

1.76 

1.40 

.87 

1.25 

1.15 

.95 

3.20* 

1.47 

.75  1.11 

27.  Equipment  Selection 

2.57 

1.32 

.82 

2.18 

0.93 

.89 

3.02* 

2.04 

.82  0.73 

28.  Installation 

1.28 

1.54 

.91 

1.17 

1.07 

.86 

0.63 

2.06 

.69  1.21 

29.  Programming 

0.83 

1.20 

.78 

0.37 

1.06 

.93 

4.82* 

1.28 

.88  0.52 

30.  Testing 

1.53 

1.60 

.86 

1.19 

1.25 

.93 

2.22* 

1.63 

.82  0.94 

31.  Operation  Monitoring 

1.69 

1.75 

.92 

1.55 

1.04 

.91 

0.72 

2.81* 

.81  1.16 

32.  Operation  and  Control 

2.05 

1.39 

.84 

2.14 

0.79 

.74 

-0:46 

3.07* 

.63  1.13 

33.  Product  Inspection 

2.31 

1.34 

.78 

2.40 

0.81 

.88 

-0.56 

2.71* 

.72  0.86 

34.  Equipment  Maintenance 

1.67 

1.63 

.92 

1.36 

1.07 

.91 

1.75 

2.32* 

.78  1.14 

35.  Troubleshooting 

2.25 

1.74 

.88 

1.48 

1.18 

.93 

4.41* 

2.17* 

.82  1.61 

36.  Repairing 

1.33 

1.49 

.92 

1.25 

1.08 

.90 

0.44 

1.89 

.73  1.03 

37.  Visioning 

2.34 

1.15 

.82 

1.88 

1.32 

.94 

2.38* 

1.32 

.59  1.43 

38.  Systems  Perception 

2.22 

1.23 

.84 

2.00 

1.25 

.92 

1.31 

1.04 

.65  1.09 

Table  3- 14a  (continued) 

Comparison  of  Incumbent  and  Analyst  Descriptive  Statistics  Across  All  Occupations  and 
Reliability  Estimates  for  Rated  Difference  Between  Jobs  for  the  Level  Scale:  Basic  and  Cross- 
Functional  Skills 


Incumbent Analyst 


Descriptor 

M 

SD 

Ik 

M 

SD 

Ik 

t 

F 

lia  d2 

39.  Identification  of  Downstream 
Consequences  2.19 

1.13 

.85 

1.82 

1.30 

.93 

2.30* 

1.32 

.71  1.00 

40.  Identification  of  Key 

Causes 

3.17 

1.17 

.86  . 

2.45 

1.22 

.93 

5.34* 

1.09 

.78  1.14 

41.  Judgment  and  Decision 

Making 

2:93 

1.33 

.89 

2.51 

1.25 

.93 

2.95* 

1.13 

.79  0.86 

42.  Systems  Evaluation 

2.02 

1.36 

.89 

1.60 

1.35 

.94 

2.79* 

1.02 

.78  0.95 

43.  Time  Management 

3.61 

1.18 

.83 

2.19 

1.19 

.93 

8.35* 

1.00 

.64  2.99 

44.  Management  of 

Financial  Resources 

1.84 

1.50 

.89 

1.37 

1.42 

.94 

2.16* 

1.12 

.62  1.78 

45.  Management  of 

Material  Resources 

1.94 

1.29 

.87 

1.99 

1.10 

.92 

-0.31 

1.37 

.69  0.89 

46.  Management  of 

Personnel  Resources  • 

2.67 

1.38 

.89 

1.56 

1.54 

.95 

5.51* 

1.24 

.68  2.59 

Incumbent  statistics  are  based  on  35  occupations  with  Skills  Questionnaire  responses  from 
at  least  4 incumbents  (mean  number  of  incumbents  — 18.51,  median  = 12,  harmonic  mean  = 
9.01).  Analyst  statistics  are  based  on  the  same  35  occupations  with  Skills  questionnaire  responses 
from  at  least  6 analysts  (mean  number  of  analysts  = 10.29,  median  = 12.0,  harmonic  mean  = 
8.66). 

The  estimate  of  reliability  was  obtained  by  calculating  the  intraclass  correlation  for  k ratings 
across  occupations:  ]CC(l,k)  = IBMS-WMS1/BMS  (Shrout  & Fleiss,  1979),  where  k is  the 
harmonic  mean  of  the  number  of  ratings  provided  on  each  occupation. 

The  t statistic  tests  for  differences  in  the  incumbent  and  analyst  group  means. 

The  F statistic  tests  for  differences  in  the  incumbent  and  analyst  group  standard  deviations. 

The  r,a  correlation  indicates  the  degree  of  relationship  between  incumbent  and  analyst  mean 
occupations  ratings. 

The  d statistic  indicates  the  squared  differences  between  incumbent  and  analyst  mean 
occupations  ratings. 

*P  < .05 


Table  3- 14b 

Comparison  of  Incumbent  and  Analyst  Descriptive  Statistics  Across  All  Occupations  and 
Reliability  Estimates  for  Rated  Difference  Between  Jobs  for  the  Level  Scale:  Basic  and  Cross- 
Functional  Skills 


Incumbent Analyst 


Descriptor 

M 

SD  Ik 

M 

SD 

Ik 

t 

F 

La  d 

1.  Reading  Comprehension 

3.65 

0.61  .83 

3.28 

0.78 

.86 

4.39*  1.65 

.77  0.38 

2.  Active  Listening 

3.63 

0.56  .78 

3.20 

0.90 

.89 

3.29*  2.61* 

.54  0.75 

3.  Writing 

3.22 

0.64  .87 

3.07 

0.82 

.86 

1.34 

1.60 

.64  0.41 

4.  Speaking 

3.54 

0.49  .75 

3.39 

1.02 

.92 

1.15 

4.39* 

.64  0.65 

5.  Mathematics 

2.96 

0.65  .74 

2.77 

0.97 

.93 

1.60 

2.26* 

.67  0.55 

6.  Science 

1.88 

0.82  .90 

1.68 

0.96 

.96 

1.99 

1.36 

.80  0.37 

7.  Critical  Thinking 

3.12 

0.70  .81 

2.48 

1.19 

.95 

4.76*  2.86* 

.75  1.05 

8.  Active  Learning 

3.01 

0.67  .79 

2.27 

1.01 

.94 

6.46*  2.27* 

;75  0.99 

9.  Learning  Strategies 

3.12 

0.50  .71 

2.11 

0.96 

.93 

9.07*  3.62* 

.76  1.46 

10.  Monitoring 

2.81 

0.65  .83 

2.69 

0.73 

.79 

1.39 

1.26 

.70  0.30 

1 1 . Social  Perceptiveness 

3.06 

0.61  .76 

2.82 

1.14 

.94 

1.69 

3.48* 

.70  0.73 

12.  Coordination 

2.99 

0.63  .75 

2.67 

1.12 

.92 

2.01 

3.12* 

.54  0.97 

13.  Persuasion 

2.65 

0.68  .85 

1.70 

0.81 

.91 

8.78* 

1.40 

.64  1.30 

14.  Negotiation 

2.62 

0.69  .84 

1.59 

0.75 

.92 

11.14* 

1.18 

.72  1.34 

15.  Instructing 

3.18 

0.57  .71 

1.99 

1.04 

.93 

8.09*  3.34* 

.55  2.15 

16.  Service  Orientation 

2.86 

0.67  .73 

2.81 

1.13 

.93 

0.30 

2.86* 

.47  0.99 

17.  Problem  Identification 

3.45 

0.62  .79 

3.04 

0.81 

.88 

3.85* 

1.70 

.65  0.54 

18.  Information  Gathering 

3.25 

0.72  .83 

2.90 

0.91 

.89 

3.73* 

1.57 

.79  0.42 

19.  Information  Organization 

2.79 

0.65  .76 

2.87 

0.82 

.87 

-0.66 

1.58 

.61  0.44 

20.  Synthesis/  Reorganization 

2.79 

0.60  .78 

2.01 

0.74 

.86 

8.33* 

1.49 

.68  0.90 

21.  Idea  Generation 

2.79 

0.69  .86 

2.09 

0.89 

.93 

6.52* 

1.68 

.71  0.88 

22.  Idea  Evaluation 

2.67 

0.72  .83 

2.14 

0.94 

.95 

4.98* 

1.71 

.75  0.65 

23.  Implementation  Planning 

2.54 

0.84  .88 

2.18 

1.04 

.92 

2.91* 

1.55 

.71  0.66 

24.  Solution  Appraisal 

2.68 

0.76  .86 

2.48 

0.80 

.87 

1.99 

1.11 

.70  0.39 

25.  Operations  Analysis 

2.06 

0.79  .82 

1.83 

0.89 

.92 

2.32* 

1.27 

.76  0.40 

26.  Technology  Design 

1.92 

0.75  .84 

1.62 

0.84 

.95 

2.61* 

1.25 

.64  0.54 

27.  Equipment  Selection 

2.36 

0.77  .81 

2.16 

0.83 

.91 

2.43* 

1.16 

.80  0.29 

28.  Installation 

1.69 

0.88  .92 

1.51 

0.81 

.90 

1.42 

1.19 

.61  0.57 

29.  Programming 

1.49 

0.78  .82 

1.16 

0.67 

.96 

5.47* 

1.36 

.89  0.23 

30.  Testing 

1.85 

0.95  .87 

1.57 

0.87 

.92 

2.46* 

1.20 

.74  0.51 

31.  Operation  Monitoring 

1.97 

0.94  .90 

1.78 

0.98 

.91 

1.56 

1.09 

.71  0.56 

32.  Operation  and  Control 

2.13 

0.80  .80 

2.27 

0.88 

.80 

-1.08 

1.21 

.56  0.63 

33.  Product  Inspection 

2.34 

0.75  .77 

2.56 

0.68 

.82 

-1.80 

1.22 

.49  0.56 

34.  Equipment  Maintenance 

1.95 

0.96  .93 

1.66 

0.94 

.94 

2.44* 

1.05 

.71  0.59 

35.  Troubleshooting 

2.27 

1.01  .88 

1.65 

0.87 

.93 

5.18* 

1.35 

.73  0.87 

36.  Repairing 

1.75 

0.85  .91 

1.47 

0.84 

.95 

2.32* 

1.01 

.64  0.58 

37.  Visioning 

2.31 

0.63  .78 

1.88 

0.90 

.92 

3.30*  2.03 

.54  0.76 

38.  Systems  Perception 

2.27 

0.68  .79 

1.87 

0.89 

.92 

3.10* 

1.73 

.56  0.71 

Table  3- 14b  (continued) 

Comparison  of  Incumbent  and  Analyst  Descriptive  Statistics  Across  All  Occupations  and 
Reliability  Estimates  for  Rated  Difference  Between  Jobs  for  the  Level  Scale:  Basic  and  Cross- 
Functional  Skills 


Incumbent Analyst 


Descriptor 

M 

SD 

Ik 

M 

SD 

Ik 

t 

F 

Da  d 

39.  Identification  of  Downstream 
Consequences  2.22 

0.65 

.84 

1.77 

0.86 

.93 

4.59* 

1.74 

.74  0.52 

40.  Identification  of  Key 
Causes 

2.74 

0.66 

.84 

2.27 

0.88 

.90 

4.39* 

1.78 

.69  0.62 

4 1 . Judgment  and  Decision 

Making 

2.70 

0.79 

.86 

2.44 

1.01 

.91 

2.25* 

1.64 

.73  0.58 

42.  Systems  Evaluation 

2.09 

0.75 

.87 

1.73 

0.87 

.92 

3.54* 

1.34 

.74  0.47 

43.  Time  Management 

3.20 

0.67 

.81 

2.25 

0.94 

.90 

7.03* 

1.96 

.55  1.53 

44.  Management  of 

Financial  Resources 

2.10 

0.90 

.87 

1.69 

0.86 

.93 

3.01* 

1.09 

.58  0.81 

45.  Management  of 
Material  Resources 

2.09 

0.73 

.85 

2.09 

0.70 

.82 

0.03 

1.12 

.58  0.41 

46.  Management  of 

Personnel  Resources 

2.57 

0.80 

.86 

1.86 

1.15 

.96 

4.69*  2.05 

.63  1.30 

^°*e‘  Incumbent  statistics  are  based  on  35  occupations  with  Skills  questionnaire  responses  from 
at  least  4 incumbents  (mean  number  of  incumbents  = 19.26,  median  = 12.5,  harmonic  mean  = 
9.01).  Analyst  statistics  are  based  on  the  same  35  occupations  with  Skills  questionnaire  responses 
from  at  least  6 analysts  (mean  number  of  analysts  = 10.29,  median  = 12.0,  harmonic  mean  = 
8.66). 

The  estimate  of  reliability  was  obtained  by  calculating  the  intraclass  correlation  for  k ratings 
across  occupations:  ICC(l,k)  = fBMS-WMSl/BMS  (Shrout  & Fleiss,  1979),  where  k is  the 
harmonic  mean  of  the  number  of  ratings  provided  on  each  occupations. 

The  t statistic  tests  for  differences  in  the  incumbent  and  analyst  group  means. 

The  F statistic  tests  for  differences  in  the  incumbent  and  analyst  group  standard  deviations. 

The  ru  correlation  indicates  the  degree  of  relationship  between  incumbent  and  analyst  mean 
occupations  ratings. 

The  d statistic  indicates  the  squared  differences  between  incumbent  and  analyst  mean 
occupations  ratings. 

*E  < .05 


Table  3-15 

Comparison  of  Incumbent  and  Analyst  Descriptive  Statistics  Across  All  Occupations  for  the 
Level  Scale  Rescored  Dichotomouslv:  Basic  and  Cross-Functional  Skills 


Incumbent Analyst 


Descriptor 

M 

SD 

Ik 

M 

SD  flc 

t 

1.  Reading  Comprehension 

0.96 

0.07 

0.50 

0.99 

0.04  0.48 

-1.80 

2.  Active  Listening 

0.92 

0.12 

0.69 

0.98 

0.05  0.24 

-2.78* 

3.  Writing 

0.91 

0.12 

0.73 

0.98 

0.07  0.66 

*3.28* 

4.  Speaking 

0.92 

0.10 

0.53 

0.98 

0.07  0.48 

-3.05* 

5.  Mathematics 

0.85 

0.17 

0.70 

0.94 

0.13  0.69 

-3.12* 

6.  Science 

0.43 

0.29 

0.87 

0.50 

0.32  0.84 

-1.70 

7.  Critical  Thinking 

0.83 

0.14 

0.62 

0.90 

0.11  0.27 

-2.83* 

8.  Active  Learning 

0.81 

0.17 

0.64 

0.90 

0.13  0.53 

-3.61* 

9.  Learning  Strategies 

0.90 

0.14 

0.54 

0.88 

0.15  0.52 

0.58 

10.  Monitoring 

0.77 

0.20 

0.81 

0.97 

0.06  0.17 

-6.87* 

1 1.  Social  Perceptiveness 

0.86 

0.15 

0.61 

0.88 

0.21  0.81 

-0.60 

12.  Coordination 

0.80 

0.17 

0.64 

0.92 

0.15  0.65 

-4.14* 

13.  Persuasion 

0.78 

0.19 

0.73 

0.75 

0.25  0.75 

0.89 

14.  Negotiation 

0.76 

0.19 

0.68 

0.68 

0.27  0.77 

2.06* 

15.  Instructing 

0.87 

0.14 

0.58 

0.73 

0.24  0.67 

3.16* 

16.  Service  Orientation 

0.78 

0.18 

0.66 

0.91 

0.20  0.83 

-3.09* 

17.  Problem  Identification 

0.89 

0.13 

0.62 

0.99 

0.03  0.31 

-4.68* 

18.  Information  Gathering 

0.90 

0.13 

0.69 

0.98 

0.07  0.66 

-4.01* 

19.  Information  Organization 

0.77 

0.18 

0.72 

0.95 

0.10  0.57 

-5.97* 

20.  Synthesis/  Reorganization 

0.82 

0.18 

0.78 

0.87 

0.18  0.70 

-2.35* 

21.  Idea  Generation 

0.80 

0.19 

0.70 

0.88 

0.12  0.40 

-3.11* 

22.  Idea  Evaluation 

0.76 

0.19 

0.74 

0.90 

0.11  0.37 

-5.25* 

23.  Implementation  Planning 

0.70 

0.22 

0.78 

0.81 

0.20  0.67 

-4.00* 

24.  Solution  Appraisal 

0.76 

0.23 

0.80 

0.95 

0.10  0.59 

-5.27* 

25.  Operations  Analysis 

0.51 

0.26 

0.78 

0.66 

0.21  0.60 

-3.69* 

26.  Technology  Design 

0.47 

0.31 

0.84 

0.60 

0.30  0.79 

-2.45* 

27.  Equipment  Selection 

0.63 

0.25 

0.78 

0.88 

0.14  0.43 

-6.71* 

28.  Installation 

0.33 

0.32 

0.87 

0.52 

0.26  0.65 

-3.86* 

29.  Programming 

0.23 

0.24 

0.63 

0.13 

0.21  0.77 

3.13* 

30.  Testing 

0.37 

0.32 

0.83 

0.55 

0.27  0.74 

-4.64* 

31.  Operation  Monitoring 

0.43 

0.32 

0.84 

0.75 

0.22  0.63 

-7.91* 

32.  Operation  and  Control 

0.58 

0.25 

0.68 

0.84 

0.12  0.00 

-7.35* 

33.  Product  Inspection ' 

0.60 

0.25 

0.71 

0.92 

0.10  0.29 

-8.25* 

34.  Equipment  Maintenance 

0.43 

0.32 

0.84 

0.65 

0.26  0.72 

-4.90* 

35.  Troubleshooting 

0.53 

0.27 

0.75 

0.68 

0.23  0.66 

-4.66* 

36.  Repairing 

0.36 

0.31 

0.83 

0.57 

0.26  0.66 

-4.67* 

37.  Visioning 

0.64 

0.20 

0.61 

0.75 

0.18  0.49 

-3.41* 

38.  Systems  Perception 

0.60 

0.21 

0.64 

0.80 

0.17  0.44 

-6.05* 

39.  Identification  of  Downstream 

0.58 

0.22 

0.72 

0.74 

0.21  0.61 

-4.26* 

Consequences 


Table  3-15  (continued) 

Comparison  of  Incumbent  and  Analyst  Descriptive  Statistics  Across  All  Occupations  for  the. 
Level  Scale  Rescored  Dichotomouslv:  Basic  and  Cross-Functional  Skills 

Incumbent Analyst 

Descriptor  M SD  a M SD  r*  t 


40.  Identification  of  Key 


Causes 

0.76 

0.20 

0.69 

0.89 

0.10 

0.10 

-4.89* 

41.  Judgment  and  Decision 

• Making 

0.74 

0.22 

0.75 

0.91 

0.10 

0.15 

-5.52* 

42.  Systems  Evaluation 

0.53 

0.26 

0.77 

0.67 

0.24 

0.67 

-3.79* 

43.  Time  Management 

0.85 

0.17 

0.75 

0.92 

0.11 

0.38 

-2.90* 

44.  Management  of 

Financial  Resources 

0.46 

0.27 

0.83 

0.52 

0.29 

0.76 

-1.09 

45.  Management  of 

Material  Resources 

0.52 

0.27 

0.84 

0.86 

0.17 

0.61 

-9.22* 

46.  Management  of 

Personnel  Resources 

0.67 

0.21 

0.71 

0.55 

0.32 

0.83 

2.19* 

Incumbent  statistics  are  based  on  35  occupations  with  Skills  questionnaire  responses  from 
at  least  4 incumbents  (mean  number  of  incumbents=  18.51,  median=12,  harmonic  mean=9.01). 
Analyst  statistics  are  based  on  the  same  35  occupations  with  Skills  questionnaire  responses  from 
at  least  6 analysts  (mean  number  of  analysts=10.29,  median=12.0,  harmonic  mean=8.66). 

The  estimate  of  reliability  was  obtained  by  calculating  the  intraclass  correlation  for  k ratings 
across  occupations:  ICC(l,k)  = rBMS-WMSl/BMS  (Shrout  & Fleiss,  1979),  where  k is  the 
harmonic  mean  of  the  number  of  ratings  provided  on  each  occupation. 

The  t statistic  tests  for  differences  in  the  incumbent  and  analyst  group  means 
*E  < .05 


Table  3-16 

Principal  Components  Analysis  Pattern  Matrix  for  the  Analyst  Level  Scale:  Basic  and  Cross- 
Functional  Skills 


Factor 


Descriptor 

FI 

F2 

Communality 

1. 

Reading  Comprehension 

.84 

.05 

.71 

2. 

Active  Listening 

.87 

-.28 

.84 

3. 

Writing 

.93 

-.03 

.86 

4. 

Speaking 

.93 

-.14 

.88 

5. 

Mathematics 

.55 

.35 

.42 

6. 

Science 

.37 

.72 

.65 

7. 

Critical  Thinking 

.93 

.24 

.92 

8. 

Active  Learning 

.92 

.28 

.92  •’ 

9. 

Learning  Strategies 

.88 

.05 

.79 

10. 

Monitoring 

.94 

.11 

.90 

11. 

Social  Perceptiveness 

.78 

-.44 

.80 

12. 

Coordination 

.94 

-.02 

.89 

13. 

Persuasion 

.89 

-.16 

.82 

14. 

Negotiation 

.89 

-.16 

.82 

15. 

Instructing 

.87 

.03 

.75 

16. 

Service  Orientation 

.60 

-.57 

.68 

17. 

Problem  Identification 

.86 

.37 

.87 

18. 

Information  Gathering 

.89 

.09 

.80 

19. 

Information  Organization 

.83 

.06 

.69 

20. 

Synthesis/Reorganization 

.88 

' .24 

.84 

21. 

Idea  Generation 

.94 

.26 

.95 

22. 

Idea  Evaluation 

.95 

.25 

.95 

23. 

Implementation  Planning 

.95 

.13 

.92 

24. 

Solution  Appraisal 

.92 

.30 

.94 

25. 

Operations  Analysis 

.16 

.52 

.85 

26. 

Technology  Design 

.30 

.80 

.74 

27. 

Equipment  Selection 

.40 

.84 

.86 

28. 

Installation 

.08 

.84 

.71 

29. 

Programming 

.31 

.48 

.32 

30. 

Testing 

.24 

.92 

.89 

31. 

Operation  Monitoring 

-.20 

.88 

.82 

32. 

Operation  and  Control 

-.06 

.78 

.61 

33. 

Product  Inspection 

.42 

.77 

.76 

34. 

Equipment  Maintenance 

-.35 

.79 

.75 

35. 

Troubleshooting 

.02 

.95 

.90 

36. 

Repairing 

-.32 

.81 

.77 

37. 

Visioning 

.95 

.19 

.94 

38. 

Systems  Perception 

.94 

.17 

.91 

39. 

Identification  of  Downstream  Consequences 

.94 

.17 

.92 

40. 

Identification  of  Key  Causes 

.93 

.21 

.90 

Table  3-16  (continued) 

Principal  Components  Analysis  Pattern  Matrix  for  the  Analyst  Level  Scale:  Basic  and  Cross- 


Functional  Skills 
Descriptor 

FI 

Factor 

F2  Communality 

41. 

Judgment  and  Decision  Making 

.94 

.14 

.90 

42. 

Systems  Evaluation 

.94 

.14 

.90 

43. 

Time  Management 

.91 

-.01 

.82 

44. 

Management  of  Financial  Resources 

.77 

-.24 

.66 

45. 

Management  of  Material  Resources 

.86 

.18 

.77 

46. 

Management  of  Personnel  Resources 

.91 

.01 

.83 

Eigenvalue 

27 

10 

Percent  of  Variance 

59.65 

21.13 

Note.  N = 35.  The  correlation  matrix  was  based  on  means  calculated  at  the  job  level.  FI  = 
Cognitive  and  Organizational  Skills,  F2  = Technical  Skills.  These  loadings  are  based  on  an 
orthogonal  varimax  rotation. 


Table  3- 17a 

Comparison  of  Level  Scale  Ratings  for  Occupations  With  Different  Educational  Requirements: 
Basic  and  Cross-Functional  Skills 


High  School  Post-Secondary 
(n  = 19) (n  = 16) 


Descriptor 

M 

SD 

M 

SD 

t 

F 

1.  Reading  Comprehension 

3.87 

0.74 

5.06 

0.56 

-5.41* 

1.71 

2.  Active  Listening 

3.81 

0.86 

4.82 

0.76 

-3.68* 

1.29 

3.  Writing 

3.14 

0.85 

4.62 

0.69 

-5.67* 

1.53 

4.  Speaking 

3.73 

0.72 

4.75 

0.47 

-5.02* 

2.33 

5.  Mathematics 

2.70 

1.08 

3.92 

0.89 

-3.64* 

1.47 

6.  Science 

0.76 

0.60 

2.44 

1.44 

-4.35* 

5.77* 

7.  Critical  Thinking 

2.96 

0.61 

4.54 

0.70 

-7.04* 

1.32 

8.  Active  Learning 

2.84 

0.71 

4.53 

0.83 

-6.38* 

1.39 

9.  Learning  Strategies 

3.33 

0.75 

4.43 

0.71 

-4.45* 

1.12 

10.  Monitoring 

2.57 

0.96 

4.08 

0.89 

-4.81* 

1.16 

11.  Social  Perceptiveness 

3.17 

0.88 

4.15 

1.08 

-2.90* 

1.51 

12.  Coordination 

2.87 

0.82 

4.40 

0.66 

-6.10* 

1.52 

13.  Persuasion 

2.49 

0.92 

4.01 

0.81 

-5.16* 

1.29 

14.  Negotiation 

2.32 

0.83 

3.58 

1.37 

-3.21* 

2.73* 

15.  Instructing 

3.24 

0.96 

4.42 

0.85 

-3.84* 

1.27 

16.  Service  Orientation 

2.88 

0.65 

3.76 

1.19 

-2.65* 

3.32* 

17.  Problem  Identification 

3.44 

0.84 

5.01 

0.59 

-6.49* 

2.00 

18.  Information  Gathering 

3.19 

0.83 

4.79 

0.55 

-6.80* 

2.32. 

19.  Information  Organization 

2.62 

0.81 

3.92 

1.00 

-4.16* 

1.51 

20.  Synthesis/  Reorganization 

2.49 

0.79 

4.13 

0.81 

-6.01* 

1.05 

21.  Idea  Generation 

2.46 

0.90 

4.35 

0.75 

-6.78* 

1.45 

22.  Idea  Evaluation 

2.29 

0.79 

4.01 

1.03 

-5.44* 

1.70 

23.  Implementation  Planning 

1.97 

0.73 

3.86 

1.28 

-5.26* 

3.09* 

24.  Solution  Appraisal 

2.17 

0.83 

4.04 

1.26 

-5.09* 

2.31 

25.  Operations  Analysis 

1.23 

0.61 

2.95 

1.45 

-4.42* 

5.57* 

26.  Technology  Design 

0.98 

0.74 

2.68 

1.45 

-4.25* 

3.79* 

27.  Equipment  Selection 

1.90 

1.02 

3.36 

1.22 

-3.81* 

1.42 

28.  Installation 

0.83 

1.13 

1.83 

1.80 

-1.93 

2.52 

29.  Programming 

0.48 

0.43 

1.25 

1.65 

-1.80 

14.80* 

30.  Testing 

0.94 

1.02 

2.23 

1.88 

-2.45* 

3.39* 

31.  Operation  Monitoring 

1.41 

1.58 

2.02 

1.93 

-1.01 

1.50 

32.  Operation  and  Control 

1.79 

1.25 

2.37 

1.51 

-1.22 

1.46 

33.  Product  Inspection 

1.81 

1.21 

2.90 

1.27 

-2.57* 

1.10 

34.  Equipment  Maintenance 

1.35 

1.40 

2.05 

1.84 

-1.25 

1.74 

35.  Troubleshooting 

1.52 

1.16 

3.11 

1.94 

-2.86* 

2.80* 

36.  Repairing 

0.89 

1.13 

1.85 

1.73 

-1.92 

2.35 

37.  Visioning 

1.91 

0.92 

2.84 

1.21 

-2.53* 

1.73 

38.  Systems  Perception 

1.66 

0.87 

2.90 

1.27 

-3.33* 

2.13 

\ J ' 
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Table  3- 17a  (continued) 

Comparison  of  Level  Scale  Ratings  for  Occupations  With  Different  Educational  Requirements: 
Basic  and  Cross-Functional  Skills 


High  School  Post-Secondary 
(n  = 19) (n  = 161 


Descriptor 

M 

SD 

M 

SD 

t 

F 

39.  Identification  of  Downstream 
Consequences 

1.64 

0.73 

2.84 

1.21 

-3.49* 

2.76* 

40.  Identification  of  Key 

Causes 

2.46 

0.88 

4.00 

0.89 

-5.15* 

1.01 

41.  Judgment  and  Decision 

Making 

2.08 

0.83 

3.94 

1.08 

-5.64* 

1.69 

42.  Systems  Evaluation 

1.30 

0.77 

2.88 

1.43 

-3.96* 

3.48* 

43.  Time  Management 

2.90 

0.85 

4.45 

0.95 

-5.05* 

1.25 

44.  Management  of  . 

Financial  Resources 

1.22 

1.09 

2.57 

1.61 

-2.83* 

2.18 

45.  Management  of 

Material  Resources 

1.41 

1.09 

2.57 

1.26 

-2.88* 

1.34 

46.  Management  of 

Personnel  Resources 

2.06 

0.96 

3.39 

1.49 

-3.07* 

2.39 

Note,  Incumbent  statistics  are  based  on  35  occupations  with  Skills  questionnaire  responses  from 
at  least  4 incumbents  (mean  number  of  incumbents  = 18.51,  median  = 12,  harmonic  mean  = 
9.01).  Nineteen  occupations  had  educational  requirements  of  high  school  or  less;  16  required 
some  post-secondary  training 

The  t statistic  tests  for  differences  in  the  low  and  high  educational  requirement  group  means. 
The  F statistic  tests  for  differences  in  the  low  and  high  educational  requirment  group  standard 
deviations. 

*B  < .05 
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Table  3- 17b 

Comparison  of  Level  Scale  Ratings  for  General  Office  Clerks  in  Organizations  Rated  as  High- 
Performance  or  Not  High-Performance:  Basic  and  Cross-Functional  Skills 

Not  High-Performance  High-Performance 


(b_e. 

m 

fn 

= 201 

Descriptor 

M 

SD 

M 

SD 

t 

F 

1.  Reading  Comprehension 

4.22 

1.52 

4.10 

1.48 

0.25 

1.05 

2.  Active  Listening 

3.89 

1.64 

3.60 

1.67 

0.54 

1.04 

3.  Writing 

3.17 

1.58 

3.45 

1.73 

-0.53 

1.20 

4.  Speaking 

2.89 

1.97 

3.95 

1.39 

-1.90 

2.01 

5.  Mathematics 

2.56 

2.12 

2.70 

1.81 

-0.22 

1.37 

6.  Science 

0.17 

0.51 

0.35 

0.99 

-0.73 

3.77* 

7.  Critical  Thinking 

2.33 

2.09 

3.30 

2.13 

-1.41 

1.04 

8.  Active  Learning 

2.72 

2.22 

2.60 

2.01 

0.18 

1.22 

9.  Learning  Strategies 

2.72 

1.71 

3.09 

2.02 

-0.61 

1.40 

10.  Monitoring 

1.67 

1.57 

2.40 

2.09 

-1.23 

1.77 

11.  Social  Perceptiveness 

2.11 

1.91 

2.95 

1.93 

-1.35 

1.02 

12.  Coordination 

2.00 

2.06 

3.05 

1.88 

-1.64 

1.20 

13.  Persuasion 

1.39 

1.82 

2.35 

2.03 

-1.54 

1.24 

14.  Negotiation 

1.00 

1.33 

2.10 

1.94 

-2.05 

2.13 

15.  Instructing 

3:62 

1.68 

3.00 

2.03 

1.03 

1.46 

16.  Service  Orientation 

2.11 

2.03 

3.55 

2.28 

-2.06* 

1.26 

17.  Problem  Identification 

3.50 

1.38 

3.80 

1.91 

-0.56 

1.92 

18.  Information  Gathering 

3.39 

1.79 

3.70 

1.72 

-0.55 

1.08 

19.  Information  Organization 

2.17 

2.31 

2.85 

2.16 

-0.94 

1.14 

20.  Synthesis/  Reorganization 

2.33 

1.14 

2.85 

1.81 

-1.06 

2.52 

21.  Idea  Generation 

1.33 

1.68 

2.10 

1.68 

-1.40 

1.00 

22.  Idea  Evaluation 

2.33 

1.53 

2.80 

1.59 

-0.72 

1.08 

23.  Implementation  Planning 

0.78 

1.40 

1.80 

1.99 

-1.85 

2.02 

24.  Solution  Appraisal 

2.11 

1.57 

2.55 

1.79 

-0.81 

1.30 

25.  Operations  Analysis 

1.00 

1.71 

1.40 

2.01 

-0.66 

1.38 

26.  Technology  Design 

0.00 

0.00 

0.35 

0.88 

-1.79 

0.00 

27.  Equipment  Selection 

1.44 

2.06 

1.85 

2.08 

-0.60 

1.02 

28.  Installation 

0.44 

1.42 

0.35 

0.93 

0.24 

2.33 

29.  Programming 

0.28 

0.75 

0.40 

1.27 

-0.36 

2.87* 

30.  Testing 

0.50 

1.54 

0.35 

1.09 

0.34 

2.00 

31.  Operation  Monitoring 

0.50 

1.47 

0.75 

1.74 

-0.48 

1.40 

32.  Operation  and  Control 

1.67 

1.41 

1.85 

1.60 

-0.38 

1.29 

33.  Product  Inspection 

0.89 

1.37 

1.95 

2.11 

-1.85 

2.37 

34.  Equipment  Maintenance 

0.61 

1.14 

1.05 

1.93 

-0.86 

2.87* 

35.  Troubleshooting 

1.00 

2.09 

1.25 

1.77 

-0.40 

1.39 

36.  Repairing 

0.39 

0.92 

0.85 

1.46 

-1.18 

2.52 

37.  Visioning 

1.61 

1.61 

1.20 

1.64 

0.78 

1.04 

38.  Systems  Perception 

1.44 

1.65 

1.35 

1.84 

0.17 

1.24 
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Table  3- 17b  (continued) 

Comparison  of  Level  Scale  Ratings  for  General  Office  Clerks  in  Organiyations  Rated  as  High- 
Performance  or  Not  High-Performance:  Basic  and  Cross-Functional  Skills 


Not  High-Performance  High-Performance 
(n  = 18) (n  = 201 


Descriptor 

M 

SD 

M 

SD 

t 

F 

39.  Identification  of  Downstream 

Consequences 
40.  Identification  of  Key 

1.00 

1.33 

0.90 

1.55 

0.21 

1.36 

Causes 

1.22 

1.73 

1.70 

1.81 

-0.83 

1.09 

41.  Judgment  and  Decision 

Making 

1.11 

1.37 

1.80 

2.09 

-1.21 

2.33 

42.  Systems  Evaluation 

0.89 

1.41 

1.25 

1.77 

-0.70 

1.58 

43.  Time  Management 

44.  Management  of 

2.61 

2.40 

3.20 

2.46 

-0.75 

1.05 

Financial  Resources 

0.78 

1.59 

1.55 

2.01 

-1.32 

1.60 

45.  Management  of 

Material  Resources 

0.22 

0.55 

1.35 

2.06 

-2.36* 

14.03* 

46.  Management  of 

Personnel  Resources 

1.22 

1.40 

1.37 

1.72 

-0.29 

1.51 

— °*e~  Statistics  are  based  on  Skills  questionnaire  responses  from  20  incumbents  in  10 
organizations  scoring  in  the  upper  quartile  on  a composite  of  organizational  performance-related 
factor  scores  (high-performance),  and  18  incumbents  in  9 organizations  scoring  in  the  lower 
quartile  (not  high-performance). 

The  t statistic  tests  for  differences  in  the  group  means. 

The  F statistic  tests  for  differences  in  the  group  standard  deviations 
*E  < .05 


Table  3- 17c 

Comparison  of  Level  Scale  Ratings  for  First  Line  Supervisors,  Clerical/ Administrative  in 
Organizations  Rated  as  High-Performance  or  Not  High-Performance:  Basic  and  Cross-Functional 
Skills 

Not  High-Performance  High-Performance 


(n  = 

15) 

(n  = 

15) 

Descriptor 

M 

SD 

M 

SD 

t 

F 

i . Reading  Comprehension 

5.33 

0.82 

4.40 

1.12 

2.61* 

1.87 

2.  Active  Listening 

5.60 

0.74 

5.27 

1.03 

1.02 

1.94 

3.  Writing 

4.86 

0.83 

5.13 

1.06 

-0.79 

1.63 

4.  Speaking 

5.13 

0.99 

4.80 

0.86 

0.98 

1.33 

5.  Mathematics 

3.47 

1.73 

3.13 

1.06 

0.64 

2.66 

6.  Science 

1.53 

1.64 

0.53 

1.36 

1.82 

1.45 

7.  Critical  Thinking 

4.67 

1.05 

4.20 

1.37 

1.05 

1.70 

8.  Active  Learning 

4.47 

1.60 

3.67 

1.76 

1.30 

1.21 

9.  Learning  Strategies 

4.20 

1.90 

4.33 

1.18 

-0.23 

2.59 

10.  Monitoring 

4.40 

1.55 

4.80 

1.26 

-0.77 

1.51 

11.  Social  Perceptiveness 

5.00 

1.07 

4.73 

1.62 

0.53 

2.29 

12.  Coordination 

4.80 

0.94 

4.40 

2.32 

0.62 

6.09* 

13.  Persuasion 

4.80 

0.77 

4.20 

1.57 

1.33 

4.16* 

14.  Negotiation 

4.60 

1.06 

3.87 

1.88 

1.31 

3.15* 

15.  Instructing 

4.81 

1.14 

3.80 

1.47 

2.10* 

1.66 

16.  Service  Orientation 

4.47 

1.46 

3.47 

2.13 

1.50 

2.13 

17.  Problem  Identification 

5.13 

1.19 

3.53 

2.07 

2.60* 

3.03* 

18.  Information  Gathering 

5.27 

1.22 

4.37 

1.34 

1.90 

1.21 

19.  Information  Organization 

4.80 

1.32 

3.20 

2.04 

2.55* 

2.39 

20.  Synthesis/  Reorganization 

4.73 

1.44 

4.13 

1.06 

1.30 

1.85 

21.  Idea  Generation 

5.00 

1.07 

4.33 

1.59 

1.35 

2.21 

22.  Idea  Evaluation 

4.67 

1.11 

3.60 

1.68 

2.05 

2.29 

23.  Implementation  Planning 

4.67 

1.23 

4.13 

1.36 

1.13 

1.22 

24.  Solution  Appraisal 

4.53 

1.19 

3.80 

2.08 

1.18 

3.06* 

25.  Operations  Analysis 

3.80 

2.31 

2.53 

2.33 

1.50 

1.02 

26.  Technology  Design 

1.87 

2.13 

1.53 

2.03 

0.44 

1.10 

27.  Equipment  Selection 

3.00 

2.36 

2.20 

2.27 

0.95 

1.08 

28.  Installation 

1.33 

1.91 

1.07 

1.98 

0.37 

1.07 

29.  Programming 

1.07 

2.05 

0.73 

1.28 

0.53 

2.57 

30.  Testing 

1.80 

2.43 

1.00 

1.69 

1.05 

2.07 

31.  Operation  Monitoring 

1.53 

2.13 

1.27 

1.83 

0.37 

1.35 

32.  Operation  and  Control 

1.80 

1.57 

2.20 

1.70 

-0.67 

1.17 

33.  Product  Inspection 

3.27 

1.94 

2.67 

2.26 

0.78 

1.36 

34.  Equipment  Maintenance 

1.33 

2.13 

1.53 

2.13 

-0.26 

1.00 

35.  Troubleshooting 

2.13 

2.26 

1.93 

2.34 

0.24 

1.07 

36.  Repairing 

1.20 

1.90 

1.47 

1.85 

-0.39 

1.05 

37.  Visioning 

3.93 

1.49 

2.67 

1.88 

2.05 

1.59 

38.  Systems  Perception 

3.73 

2.12 

3.47 

2.29 

0.33 

1.17 
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Table  3- 17c  (continued) 

Comparison  of  Level  Scale  Ratings  for  First  Line  Supervisors.  Clerical/Administrative  in 

Organizations  Rated  as  High-Performance  or  Not  High-Performance:  Basic  and  Cross-Functional 
Skills 


Not  High-Performance  High-Performance 
(d=  15) (n  = 151 


Descriptor 

M 

SD 

M 

SD 

t 

F 

39.  Identification  of  Downstream 

Consequences 

4.07 

1.98 

3.20 

2.24 

1.12 

1 28 

40.  Identification  of  Key 

Causes 

4.87 

1.41 

3.87 

1.68 

1.77 

1.47 

41.  Judgment  and  Decision 

Making 

5.13 

1.41 

4.00 

1.65 

2.03 

1.37 

42.  Systems  Evaluation 

4.60 

1.99 

2.53 

1.85 

2.95* * 

1.16 

43.  Time  Management 

44.  Management  of 

5.13 

1.19 

5.07 

1.83 

0.12 

2.36 

Financial  Resources 
45.  Management  of 

4.00 

2.56 

3.13 

2.10 

1.01 

1.49 

Material  Resources 

3.93 

2.15 

2.60 

1.76 

1.85 

1.49 

46.  Management  of 

Personnel  Resources 

5.33 

1.59 

5.27 

1.44 

0.12 

1.22 

Note.  Statistics  are  based  on  Skills  questionnaire  responses  from  15  incumbents  in  10 
organizations  scoring  in  the  upper  quartile  on  a composite  of  organizational  performance-related 
factor  scores  (high-performance),  and  15  incumbents  in  8 organization*;  scoring  in  the  lower 
quartile  (not  high-performance). 

The  t statistic  tests  for  differences  in  the  group  means. 

The  F statistic  tests  for  differences  in  the  group  standard  deviations 

*E  < .05 
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Evidence  for  the  Reliability  and  Validity  of  the  Measures 


David  P.  Costanza 
Edwin  A.  Fleishman 
Joanne  C.  Marshall-Mies 
Management  Research  Institute,  Incorporated 


Occupational  knowledge  is  an  important  element  in  the  understanding  and  description  of 
virtually  all  occupations  in  the  workplace.  The  study  of  occupational  knowledge  can  provide 
important  information  for  a variety  of  activities  in  the  domain  of  work,  including  training,  career 
counseling,  selection  and  person/job  matching.  While  knowledge  is,  by  definition,  domain- 
specific,  the  job-relevant  knowledges  taxonomy  was  developed  to  be  domain-specific  yet  still 
broad  enough  to  be  useful  in  describing  and  categorizing  a variety  of  different  occupations.  This 
chapter  briefly  describes  the  development  of  a taxonomy  of  job-required  knowledges  and  its 
associated  measurement  system  as  well  as  the  results  from  the  initial  0*NET  data  collection 
effort.  In  this  chapter,  we  begin  by  briefly  reviewing  the  development  of  the  knowledges 
taxonomy  and  its  measures.  (A  complete  account  of  the  conceptual  background,  previous 
research  on  the  knowledge  domain,  and  the  development  process  has  been  presented  in  a 
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previous  report;  see  Fleishman,  Costanza,  Marshall-Mies,  Wetrogan,  Uhlman,  1995).  Next,  we 
describe  the  results  obtained  with  the  knowledge  taxonomy  and  measurement  system,  when  these 
were  used  to  describe  occupations  in  the  present  0*NET  study.  Finally,  we  discuss  some  of  the 
implications  of  these  results  on  further  development  of  the  0*NET  system. 

Knowledge  is  defined  as  a collection  of  discrete  but  related  facts,  information,  and 
principles  about  a certain  domain.  Knowledge  is  acquired  through  formal  education  or  training 
or  accumulated  through  specific  experiences.  The  fact  that  these  pieces  of  information  are 
organized  into  a coherent  structure  is  critical  to  this  definition.  Further,  some  knowledges  are 
more  general  than  others  in  that  they  are  important  to  successful  performance  in  a greater  variety 
of  jobs  in  the  economy.  Other  knowledges  are  more  specific  and  apply  to  a narrower  range  of 
occupations. 

Therefore,  in  developing  a taxonomy  of  knowledges  for  describing  job  requirements,  it 
was  important  to  deal  with  the  issue  of  the  specificity  level  needed  to  provide  a comprehensive 
but  parsimonious  taxonomic  system.  Ideally,  one  would  strive  to  make  uniform  the  level  of 
specificity  of  the  knowledge  constructs,  so  that  they  are  broad  enough  to  cover  multiple  domains, 
but  not  so  encompassing  that  they  are  of  limited  use  as  components  of  a knowledge  taxonomy. 

Background 

The  knowledges  used  in  the  0*NET  data  collection  effort  evolved  from  an  earlier  effort 
by  Costanza  and  Fleishman  (1992)  that  attempted  to  describe  and  measure  a wide  variety  of  job- 
relevant knowledge  areas.  The  intent  of  this  effort  was  to  identify,  define,  classify,  and  create  a 
measurement  system  for  job  knowledge  areas  that  in  turn  could  be  used  in  any  of  a variety  of  job- 
related  activities.  The  original  development  effort  and  the  additional  steps  taken  to  adapt  the 
knowledge  scales  for  the  present  effort  are  summarized  below. 
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Taxonomy 

In  the  previous  work  by  Costanza  and  Fleishman  (1992),  original  knowledges  were 
developed  using  a four  step  process.  First,  a broad  literature  search  was  conducted  to  find 
previously  identified  job  knowledges.  Unfortunately,  much  of  the  literature  has  focused  on  the 
nature  of  the  structures,  and  the  processes  involved  in  developing  and  analyzing  knowledge, 
rather  than  on  taxonomies  of  knowledges  themselves.  While  several  existing  taxonomies  (e.g., 
Prediger,  1989;  Peterson,  1992)  provided  some  initial  possible  candidate  knowledges,  it  was 
apparent  that  an  empirical  approach  to  developing  the  taxonomy  might  be  more  appropriate  and 
pragmatic.  Accordingly,  the  approach  of  analyzing  descriptions  from  as  many  occupations  as 
possible  and  looking  for  tasks  and/or  behaviors  that  were  representative  of  underlying 
knowledges  was  adopted.  Not  surprisingly,  a particularly  useful  source  of  occupation 
descriptions,  in  which  the  most  occupations  were  explicitly  identified  and  described,  was  the 
Department  of  Labor's  Dictionary  of  Occupational  Titles  (DOT;  Department  of  Labor,  1991). 

Therefore,  the  second  step  was  to  thoroughly  review  the  DOT  and  to  extract  from  the 
occupation  descriptions  an  initial  list  of  job  knowledges.  Each  occupation  description  in  the  DOT 
was  read  and  examined  and  any  knowledges  that  were  listed  or  implied  in  the  tasks  described 
were  extracted.  This  analysis  of  the  DOT  occupation  descriptions  yielded  68  qualitatively 
different  knowledges.  Other  research  efforts  (e.g.,  Prediger,  1989;  McKinney  & Greer,  1985; 
Campbell,  Ford,  Rumsey,  Pulakos,  Borman,  Felker,  De  Vera,  & Riegelhaupt,  1990;  Fleishman, 
1992)  were  again  reviewed  to  identify  additional  knowledges,  and  candidates  were  added  to  the 
list.  The  list  was  checked  for  omissions,  ambiguities,  or  redundancies  and  combined  into  a 
preliminary  taxonomy  of  86  knowledges.  Once  the  list  of  knowledges  had  been  developed. 
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prototype  task  examples  indicating  high,  medium,  and  low  amounts  of  the  knowledge  were 
generated  using  the  occupation  descriptions  from  which  the  knowledges  came. 

At  this  stage,  it  became  clear  that  the  level  of  specificity  still  varied  somewhat  across  the 
knowledges.  Further,  it  appeared  that  the  knowledges  seemed  to  be  grouped  around  several 
broader,  superordinate  areas.  Hence,  the  third  step  in  the  development  process  consisted  of  a 
search  to  identify  existing  taxonomies  of  job  families  or  job  groups  into  which  the  knowledges 
could  be  categorized.  By  grouping  the  knowledges  into  larger  categories  based  on  similarity,  the 
specificity  issue  could  be  addressed  to  improve  the  usefulness  of  the  taxonomy.  This  effort 
yielded  seven  superordinate  categories— artistic/creative,  business/administrative,  mechanical/ 
skilled  trades,  outdoor  work,  professional,  scientific,  and  service  sector-which  appeared  to 
capture  many  of  the  proposed  knowledges.  The  initial  knowledges  were  sorted  into  these  seven 
categories  and  again  reviewed  for  completeness,  ambiguity,  and  reasonableness.  As  a result  of 
this  review,  the  knowledges  were  consolidated  into  52  knowledges. 

Step  four  consisted  of  the  identification  and  development  of  task  examples  to  help  in  the 
rating  of  the  levels  of  the  job  knowledges  required  for  different  occupations  and  tasks.  The 
format  for  developing  rating  scales  for  the  52  knowledges  followed  that  developed  by  Fleishman 
(1975a,  b,  1992),  for  rating  ability  requirements  (Fleishman  & Quaintance,  1984),  and  Fleishman 
& Mumford  (1988).  The  list  of  sample  tasks  developed  during  the  DOT  review  was  revised  and 
shortened  and  a survey  of  the  tasks  for  the  52  remaining  knowledges  was  developed.  Nineteen 
raters  completed  a survey  in  which  they  were  asked  to  make  two  determinations  about  each  task 
for  a given  knowledge:  1)  if  the  task  required  any  amount  of  the  knowledge;  and,  if  so,  2)  on  a 
scale  of  1 to  7,  how  much  of  the  knowledge  is  required  for  performance  of  the  task.  The  results 
of  this  rating  process  yielded  high  reliabilities  (.89  to  .98),  a good  range  of  means  with  low 
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standard  deviations,  allowing  for  accurate  task  anchoring,  and  a consensus  that  several  of  the 
knowledges  should  be  combined.  This  resulted  in  the  final  list  of  49  knowledges  (two  were 
combined  and  one  was  deleted  based  on  rater  feedback).  Based  on  the  task  ratings,  anchors  with 
high  reliabilities,  appropriate  (high,  medium,  low)  means  and  small  standard  deviations  were 
chosen  to  anchor  each  of  the  scales.  The  scales  were  incorporated  into  the  Fleishman  Job 
Analysis  Survey  (F-JAS)  (Fleishman,  1992;  Costanza  & Fleishman,  1992)  as  part  of  this  job 
analysis  system. 

At  this  stage  in  their  development,  the  49  knowledge  scales  from  the  F-JAS  were  used  in 
several  efforts  to  describe  and  understand  job  performance.  One  study  of  75  occupations 
involved  18  of  the  knowledge  scales  (Hauke,  Costanza,  Baughman,  Mumford,  Stone,  Threlfall, 
& Fleishman,  1995).  In  this  effort,  a major  governmental  agency  was  interested  in  validating  the 
key  selection  measures  used  by  the  agency  for  entry-level  positions.  The  inclusion  of  the 
knowledge  scales  substantially  improved  the  prediction,  overall  quality,  and  parsimony  of  both 
the  occupation  description  and  of  an  effort  to  cluster  occupations  according  to  ability  and 
knowledge  requirements.  In  another  effort,  the  knowledge  scales  were  used  in  a study  of  several 
State  Police  jobs  (Trooper,  Corporal,  Sergeant,  Lieutenant).  Again,  interrater  reliabilities  for 
knowledge  profiles  were  from  .90  to  .95  when  23  raters  were  used  (Wetrogan,  Uhlman,  & 
Fleishman,  1995)  and  the  knowledge  profiles  obtained  helped  differentiate  the  occupations. 
Based  on  the  above,  it  was  concluded  that  the  initial  development  effort  had  produced  a 
potentially  reliable  and  useful  taxonomy  for  describing  and  analyzing  occupation  requirements. 

Sample  and  Measures 

For  the  present  project,  the  49  knowledge  scales  were  initially  pre-tested  by  DOT  project 
staff  on  a sample  of  job  incumbents  from  approximately  30  occupations.  This  pilot 
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administration  and  subsequent  feedback  from  the  Occupational  Analysis  Field  Centers  (OAFCs) 
found  that  while  the  scales  evidenced  sufficient  reliability,  there  was  room  for  improvement  in 
the  taxonomy  and  rating  scales.  Accordingly,  the  knowledges  were  systematically  reviewed  and 
edited,  and  the  knowledges,  the  measurement  scales,  and  the  instructions  for  completing  these 
scales  were  also  revised  to  maximize  their  usefulness  for  the  present  effort  and  to  improve  their 
overall  reliability  and  validity.  To  accomplish  this,  the  extant  literature  was  again  reviewed,  the 
knowledges,  knowledge  clusters,  and  rating  scales  were  revised,  as  were  the  rating  scales,  and 
knowledge  specialty  areas  were  identified  and  incorporated  into  the  scales.  These  steps  are 
briefly  described  below. 

Several  new  sources  in  the  literature  were  identified,  although  as  before,  they  did  not 
provide  a direct  list  of  knowledges.  Among  these  sources  were  the  SCANS  list  of  skills  needed 
for  employment  (Peterson,  1992)  and  the  list  of  competencies  in  the  Multipurpose  Occupational 
Systems  Analysis  Inventory-Close-ended  (MOSAIC)  (Corts  & Gowing,  1992).  While  these 
systems  are  broader  rn  definition  than  is  desired  for  the  current  effort,  a mapping  of  these  skills 
and  competencies  against  the  knowledge  taxonomy  confirmed  that  no  major  knowledges  had 
been  omitted  in  the  taxonomy. 

Next,  the  knowledges  and  knowledge  clusters  were  revised.  The  goal  was  to  develop  a 
more  parsimonious  set  of  knowledges  classified  into  broader  clusters  using  the  pilot  data  as  a 
gurde.  One  means  of  establrshing  the  content  validity  of  the  knowledge  classification  scheme  was 
to  compare  the  original  classification  with  that  created  using  an  independent  methodology.  An 
appropriate  and  convenient  comparison  was  to  the  National  Occupational  Information 
Coordinating  Committee  (NOICC)  hierarchical  clustering  of  244  National  Units  of  Analysis 
(NUAs).  This  clustering  hierarchy  provides  a mechanism  for  matching  job  market  demand  and 
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institutional  supply  data  gathered  at  the  state  level  by  State  Occupational  Information 
Coordinating  Committees  (SOICC). 

Although  there  were  some  differences,  the  NOICC  data  and  the  knowledges  taxonomy 
were  similar  in  terms  of  the  numbers  and  content  of  knowledge  clusters  and  knowledges.  A 
direct  comparison  of  the  knowledges  and  NOICC  structures  provided  additional  information 
related  to  the  comprehensiveness  and  level  of  specificity  of  the  knowledges.  Each  of  the 
knowledges  was  mapped  onto  the  NOICC  groups.  This  effort  demonstrated  that  the  knowledges 
covered  all  areas  covered  by  the  NOICC  system.  The  differences  between  the  two  that  did 
manifest  themselves  were  carefully  reviewed.  Based  on  this  evaluation,  the  information  gleaned 
from  the  new  literature  review,  and  the  desire  to  reduce  demands  on  raters,  several  of  the  49 
knowledges  were  combined.  The  result  was  the  final  list  of  33  knowledges.  These  knowledges 
are  listed  in  Figure  4-1.  Within  each  of  these  33  knowledges,  job  specialty  requirements  were 
identified  and  listed. 

Responses  from  the  pilot  administration  also  indicated  that  the  knowledges  differed  in 
their  applicability  across  occupations.  That  is,  some  of  the  knowledges  appeared  to  be  applicable 
to  a broader  range  of  occupations,  whereas  other  knowledges  seemed  to  apply  to  a narrower 
range  of  occupations.  Therefore,  a decision  was  made  to  organize  the  scales  in  a more 
meaningful  way.  Another  review  and  ratings  by  an  independent  panel  led  to  a rearrangement  of 
the  scales,  such  that  more  general  cross-occupational  knowledges  appear  first,  followed  by  the 
more  occupation-specific  knowledges.  This  ensured  that  the  raters  would  be  more  likely  to 
encounter  knowledges  relevant  to  their  jobs  earlier  in  the  survey. 

In  revising  the  scales,  the  improvements  in  the  rating  scales  were  pursued  while  trying  to 
maintain  the  integrity  of  the  original  development  process.  Another  independent  panel  reviewed 
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and  then  edited  each  knowledge  rating  scale  including  the  knowledge  definition,  the  high  and 
low  level  descriptors,  and  the  task  anchors.  In  addition  to  revising  the  knowledge  scale  content, 
the  editing  process  was  designed  to  increase  the  scales’  clarity  and  make  the  reading  level  more 
appropriate  for  incumbents  whose  jobs  require  less  demanding  reading  levels  and  cognitive 
skills.  Scale  anchors  were  checked  and,  if  necessary,  replaced  to  make  them  less  esoteric  and 
more  readily  identifiable  by  different  incumbent  populations.  Other  anchors  were  reviewed  to 
ensure  that  they  reflected  sufficient  amounts  of  the  knowledge  required  and  did  not  appear  trivial 
to  job  incumbents.  This  completed  the  revision  and  modification  process  of  the  knowledge  scales 
for  use  in  the  0*NET  data  collection.  As  noted  above,  job  knowledge  specialty  areas  were 

identified  and  administered  to  the  job  incumbents,  but  those  data  were  not  analyzed  at  this  stage 
in  the  project. 

Both  incumbents  and  analysts  were  asked  to  make  two  ratings  for  each  of  the  job 
knowledges.  First,  they  were  asked  to  rate  the  level  of  the  knowledge  required  for  the  job,  using  a 
one  to  seven  scale.  They  were  given  a “Does  not  apply”  option  whereby  they  could  rate  a 
particular  knowledge  zero  if  no  level  was  required.  Second,  the  raters  were  asked  to  rate  the 
importance  of  the  knowledge  for  job  performance,  using  a one  to  five  scale.  If  the  raters 
responded  “zero”  to  the  level  question,  they  were  instructed  to  skip  the  importance  rating.  The 
ratings  were  gathered  using  the  scaling  format  developed  earlier  by  Fleishman  (1975b)  and 
described  elsewhere  (Fleishman  & Quaintance,  1984;  Fleishman  & Mumford,  1991;  Fleishman 
et  al.,  1995).  For  each  knowledge,  the  scale  provided  a construct  definition,  clarification 
statements  defining  high  and  low  knowledge  levels  requirements,  and  task  anchors  (as  developed 
and  described  above)  at  different  points  on  the  scale.  An  example  of  one  of  the  knowledge  rating 
scales  is  presented  in  Figure  4-2.  Note  that  respondents  were  asked  to  indicate  the  relevance  of 
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each  of  several  job  specialty  requirements  if  they  had  rated  the  general  knowledge  area  as 
relevant;  however,  those  data  are  not  analyzed  here. 

In  the  initial  data  collection  effort,  two  sets  of  judges  were  used  to  provide  ratings  on  the 
knowledges  for  a variety  of  occupations.  The  first  sample  included  job  incumbents  from  80 
occupations.  These  occupations  were  selected  using  a stratified  random  sampling  procedure 
designed  to  represent  over  770  occupations  and  45%  of  the  employed  U.S.  population.  Because 
of  the  desire  for  reliability  and  stability  of  the  ratings,  only  occupations  for  which  at  least  four  (4) 
incumbents  provided  ratings  were  used.  This  criteria  resulted  in  a final  sample  of  649  incumbents 
from  33  of  the  occupations.  These  33  occupations  are  listed  in  Figure  4-3. 

The  second  set  of  judges  consisted  of  occupational  analysts  provided  by  the  OAFCs.  In 
this  sample,  at  least  six  analysts  were  asked  to  rate  the  knowledge  requirements  for  each  of  the  80 
occupations.  Because  of  the  limited  number  of  occupations  represented  in  the  first  sample, 
analyses  were  carried  out  only  on  the  analysts’  ratings  for  the  same  33  occupations  represented  in 
the  incumbent  sample.  The  final  number  of  analysts  ratings  was  324,  although  in  many  cases, 
analysts  rated  more  than  one  occupation  each. 

Results 

Descriptive  Statistics 

Table  4-1  presents  the  means,  standard  deviations,  standard  error  of  measurements,  and 
interclass  correlations  for  both  the  level  and  importance  scales  for  the  33  knowledges.  Because 
these  results  are  averaged  across  33  occupations,  only  broad  patterns  of  responses  can  be 
interpreted.  For  example,  the  knowledges  with  the  three  highest  level  ratings.  Clerical 
(Descriptor  #2)  (M  = 3.30),  English  Language  (Descriptor  #24)  (M  = 3.18),  and  Mathematics 
(Descriptor  #14)  (M  = 2.93),  are  all  more  general  knowledge  areas  that  might  apply  to  a wide 
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variety  of  government  or  public  service  jobs.  These  three  were  also  rated  the  highest  of 
importance  (M  = 2.83,  2.98,  and  2.82  respectively).  On  the  other  hand,  those  knowledges 
receiving  the  lowest  ratings  were  some  of  the  more  esoteric  ones  such  as  Fine  Arts  (Descriptor 
#26)  (M  = .58),  Food  Production  (Descriptor  #8)  (M  = .68),  and  History  and  Archeology 
(Descriptor  #27)  (M  = .71)  that  might  be  expected  to  be  necessary  for  only  a few  specific 
occupations.  These  results  provide  the  first  evidence  that  the  knowledges  were  acting  as 
intended;  that  is,  tapping  relatively  domain-specific  knowledge  areas  that  might  help  better 
differentiate  among  occupations.  This  point  is  further  supported  by  the  relatively  high  standard 
deviations  for  the  seven-point  level  and  five  point  importance  scale,  indicating  a broad  range  of 
responses  for  most  of  the  knowledges.  The  average  SD  for  level  was  almost  two  (SD  = 1.94)  and 
for  importance  was  over  one  (SD  =1.11). 

Reliability 

The  intraclass  correlations  (reliabilities)  in  Table  4-1  reflect  the  general  agreement  found 
among  the  job  incumbents  across  occupations.  The  majority  of  the  reliabilities  for  level  and 
importance  were  above  .80  (minimum  = .37,  maximum  = .95)  and  those  that  exhibited  unusually 
low  reliabilities  were  also  the  same  ones  that  had  the  lowest  mean  ratings,  such  as  Food 
Production  (Descriptor  #8)  (&  = .54,  level)  and  Fine  Arts  (Descriptor  #26)  (&  = .63,  importance). 
Given  these  relatively  high  overall  reliabilities,  it  is  not  surprising  that  the  standard  error  of 
measurement  results  were  reasonable,  with  almost  all  being  below  1 .0  for  level  ratings  and  .75 
for  importance  ratings. 

Table  4-2  shows  the  reliability  estimates  for  each  knowledge  and  rating,  for  one  and  30 
raters.  The  results  here  are  consistent  with  the  observed  (k-rater)  reliabilities;  one-rater 
reliabilities  are  mostly  in  the  .20s  and  .30s,  the  exceptions  being  those  relatively  low-rated 
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knowledges  such  as  Foreign  Language  (Descriptor  #25)  (&  = .05).  The  reliabilities  for  30  raters 
were  consistently  in  the  .80s  and  .90s. 

Scoring 

In  an  effort  to  assess  whether  the  scale  itself  had  any  impact  on  the  ratings,  reliability 
estimates  were  calculated  on  the  incumbent  data  for  three  variations  of  the  level  scale:  one  (low) 
to  seven  (high),  zero  (not  relevant)  to  seven  (high),  and  zero  (not  relevant!  to  one  (relevant).  The 

I 

results  of  this  analysis  are  presented  in  Table  4-3.  In  general  the  reliabilities  were  relatively  high 
regardless  of  the  scale  type  used.  However,  one  interesting  finding  from  this  analysis  was  that  for 
several  of  the  knowledges  with  especially  low  mean  ratings  and  reliabilities  (e.g..  Food 
Production  [Descriptor  #8],  Foreign  Language  [Descriptor  #25],  and  Fine  Arts  [Descriptor  #26]), 
the  reliabilities  increased  substantially  when  the  alternate  scales  were  used.  For  example,  the 
reliability  of  the  level  ratings  on  Food  Production  (Descriptor  #8)  was  .54  with  the  zero  to  seven 
scale,  but  .81  with  one  to  seven  and  was  still  .71  with  zero-one.  It  appears  that  the  raters  were  in 
relatively  more  disagreement  as  to  whether  or  not  these  knowledges  were  relevant  at  all  than 
about  their  appropriate  level,  if  they  thought  it  was  relevant.  This  suggests  that  we  might  need  to 
be  cautious  in  the  use  of  these  less  common  knowledge  areas  given  their  somewhat  lower 
reliabilities. 

Analyses  of  Variance 

Table  4-5  shows  the  overall  agreement  among  incumbents,  when  calculated  across  all 
occupations  and  knowledges,  based  on  the  analysis  of  variance  (ANOVA)  results  shown  in 
Tables  4-4a  and  4-4b.  This  reliability  was  similar  for  both  level  and  importance  ratings  (r^  = .85 
and  .86,  respectively).  Again,  the  Spearman-Brown  formula  was  used  to  estimate  the  reliability 
for  one  and  30  raters  with  the  results  for  both  scale  types  also  similar-.40  and  .39  for  one  rater 
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and  .95  and  .94  for  30  raters.  Overall,  these  numbers  suggest  that  given  a sufficient  number  of 
raters,  all  of  the  knowledge  scales  can  be  reliably  rated  by  incumbents  from  a wide  variety  of 
occupations. 

The  ANOVA  results  from  Tables  4-4a  and  4-4b  add  further  indications  that  the 
knowledges  do  serve  to  help  differentiate  occupations.  Both  level  and  importance  demonstrated 
significant  differences  between  and  within  occupations  across  knowledges.  For  example,  the 
level  scale  showed  significant  differences  across  occupations  (F  = 3.79,  p < .05),  within 
occupations  across  knowledges  (F  = 66.50,  p < .01),  and  for  the  interaction  between  the  two  (F  = 
6.90,  p < .05).  The  importance  scale  evidenced  similar  results  with  differences  across 
occupations  (F  = 3.46,  p < .05),  across  knowledges  within  occupations  (F  = 70.91,  p < .01),  and 
the  interaction  between  the  two  (F  = 6.74,  p < .05),  all  significant.  This  pattern  continued  when 
the  knowledges  were  aggregated  to  the  higher  level  of  10  broad  groupings  by  using  the  simple 
mean  scores  on  the  33  descriptors  combined  into  the  ten  higher-order  categories  (see  Figure  4-1). 
Tables  4-6a  and  4-6b  show  that,  once  again,  both  the  level  and  importance  scales  showed 
significant  differences  between  and  within  occupations  and  for  the  interaction  between 
occupations  and  knowledges.  Table  4-7  shows  that  the  reliabilities  for  these  higher  level 
groupings  are  similar  to  those  for  the  individual  scales. 

Descriptor  and  Scale  Relationships 

In  order  to  further  assess  what  relationships  existed  among  the  knowledge  areas, 
correlations  were  run  across  occupations  for  both  the  level  and  importance  scales  at  both  the 
occupational  and  individual  levels.  A summarization  of  the  correlations  among  the  scales  across 
descriptors  and  across  occupations  is  presented  in  Table  4-8.  These  mean  correlations  are 
substantial.  .95  for  descriptors  across  occupations  and  .90  for  occupations  across  descriptors. 
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The  correlations  of  descriptors  within  scales  are  presented  in  Tables  4-9a,  4-9b,  4-lOa, 
and  4- 10b.  While  the  33  by  33  matrices  in  these  tables  are  a bit  overwhelming,  some  interesting 
findings  shine  through.  The  occupation-level  analysis  in  Table  4-9a  of  the  level  scale  data  shows 
that  Administration  and  Management  (Descriptor  #1)  was,  as  expected,  highly  correlated  with  the 
other  knowledges  in  its  a priori  “cluster”  of  Business  and  Management,  including  Economics  and 
Accounting  (Descriptor  #3)  (r  = .66 ) and  Personnel  and  Human  Resources  (Descriptor  #6)  (r  = 
.84),  as  well  as  with  other  conceptually  related  areas  such  as  English  Language  (Descriptor  #24) 

(r  = .71),  necessary  for  managerial  communication,  and  Legal,  Government,  and  Jurisprudence  (r 
= .52).  There  also  appeared  to  be  an  engineering  “group”  with  Engineering  and  Technology 
(Descriptor  #10)  highly  correlated  with  Design  (Descriptor  #1 1)  (r  = .80),  Building  and 
Construction  (Descriptor  #12)  (r  = .75),  and  Mathematics  (Descriptor  #14)  (r  = .59).  Several 
other  groupings  emerged  including  one  representing  the  Arts  and  Humanities  cluster,  where 
Foreign  Language  (Descriptor  #25)  and  Fine  Arts  (Descriptor  #26)  were  highly  related  (r  = .76) 
as  were  English  Language  (Descriptor  #24)  and  Philosophy  and  Theology  (Descriptor  #28)  (r  = 
.69).  Overall,  the  level  correlations  seemed  to  support  the  a priori  clusters  grouping  related 
knowledges  together.  The  importance  scale  correlations  were  less  systematic,  although  there 
were  some  similar  patterns  of  correlations,  especially  in  the  Engineering  and  Technology 
(Descriptor  #10)  cluster  and  the  Math  and  Science  cluster. 

The  second  interesting  finding  here  is  that  there  were  almost  no  significant  negative 
relationships  among  the  knowledges.  For  example,  one  might  have  expected  that  when  a job 
requires  a very  high  level  of  Mathematical  or  Mechanical  (Descriptor  #13)  knowledge,  that  other 
areas  such  as  Fine  Arts  (Descriptor  #26)  or  Philosophy  and  Theology  (Descriptor  #28)  might  be 
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rated  significantly  lower.  In  fact,  these  relationships,  as  well  as  others  like  them,  were  generally 
in  the  .05  to  . 10  range,  non-significant  to  be  sure,  but  not  negative  either. 

Tables  4- 10a  and  4- 10b  present  the  individual-level  correlations,  computed  on  a sample 
of  four  incumbents  from  each  occupation  (in  order  to  balance  the  impact  of  each  occupation  on 
the  matrix).  Although  the  focus  of  the  analyses  reported  in  this  chapter  is  on  the  usefulness  ahd 
accuracy  of  the  measure  for  occupational  description,  these  tables  are  included  for  the  reader’s 
convenience. 

Factor  Structure 

A factor  analysis  was  conducted  on  the  knowledges,  both  to  see  if  the  a priori  groupings 
would  be  repeated  empirically  and  to  add  additional  information  about  the  interpretation  of  the 
occupation  description  process.  The  knowledges  were  entered  into  a principal  components 
analysis  and  the  resulting  scree  plot  and  eigenvalues  suggested  that  seven  components/factors 
should  be  retained.  These  can  be  seen  in  Table  4-11.  While  there  were  fewer  factors  than  a priori 
clusters,  the  final  components  were  very  clean  and  inteipretable.  The  first  factor  was  marked  by 
Geography  (Descriptor  #20)  (r  = .88),  Fine  Arts  (Descriptor  #26)  (r  = .86),  History  and 
Archeology  (Descriptor  #27)  (r  = .84),  English  Language  (Descriptor  #24)  (r  = .55),  and  Foreign 
Language  (Descriptor  #25)  (r  = .74),  among  others.  Accordingly,  it  was  called  arts  and 
humanities.  The  second  factor  was  equally  identifiable  and  was  termed  science  and  technology 
by  virtue  of  high  loadings  on  Mechanical  (Descriptor  #13)  (r  = .93),  Engineering  and  Technology 
(Descriptor  #10)  (r  = .92),  Physics  (Descriptor  #15)  (r  = .74),  Chemistry  (Descriptor  #16)  (r  = 
.63),  along  with  several  other  related  knowledge  areas.  The  principle  knowledges  for  the  third 
factor  were  Public  Safety  and  Security  (Descriptor  #29)  (r  = .86)  and  Law,  Government,  and 
Jurisprudence  (Descriptor  #30)  (r  = .86),  hence  making  it  a law  enforcement  factor.  Additionally, 
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several  other  knowledges  that  might  reasonably  be  expected  to  contribute  to  law  enforcement 
also  showed  high  loadings.  These  included  elements  of  this  work  such  as  Sociology  and 
Anthropology  (Descriptor  #19)  (r  = .62),  Psychology  (Descriptor  #18)  (r  = .68 ),  and  Therapy  and 
Counseling  (Descriptor  #22)  (r  = .44).  This  echoes  the  findings  reported  earlier  regarding  the 
mean  level  ratings  for  certain  knowledges  among  Police  Patrol  Officers  (see  Table  4-12^). 

The  fourth  factor  was  a clerical  factor  and  was  marked  by  high  loadings  for  Clerical 
(Descriptor  #2)  (r  = .81),  English  Language  (Descriptor  #24)  (r  = .67),  Computers  (r  = .56),  and 
Administration  and  Management  (Descriptor  #1)  (r  = .56),  all  relevant  to  various  aspects  of 
Clerical  (Descriptor  #2)  work.  The  next  factor  was  also  clearly  identifiable  by  its  markers 
including  Biology  (Descriptor  #17)  (r  = .94),  Medicine  and  Dentistry  (Descriptor  #21)  (r  = .81), 
and  Chemistry  (Descriptor  #16)  (r  = .72)  as  a medical  factor.  Interestingly,  Education  and 
Training  (Descriptor  #23)  also  loaded  relatively  highly  on  the  factor  (r  = .45),  suggesting  that  the 
medical  personnel  surveyed  may  have  had  some  educational  responsibilities  as  part  of  their  jobs. 
Factor  six  was  similar  to  factor  four,  except  that  it  seemed  to  represent  broader  areas  of 
administrative  and  managerial  activities  and  therefore  was  named  business  administration. 
Knowledge  areas  here  included  Sales  and  Marketing  (Descriptor  #4)  (r  = .94),  Customer  and  . 
Personal  Service  (Descriptor  #5)  (r  = .71),  Economics  and  Accounting  (Descriptor  #3)  (r  = .57 ), 
Personnel  and  HR  (r  = .55),  along  with  Administration  and  Management  (Descriptor  #1)  (r  = 

.50).  The  final  factor  was  a little  unusual  with  its  one  very  high  positive  loading.  Computers  and 
Electronics  (Descriptor. #9)  (r  = .65),  and  one  very  high  negative  loading.  Food  Production 
(Descriptor  #8)  (r  = -.77).  Despite  changes  in  technology.  Food  Production  (Descriptor  #8),  the 
growing,  producing,  and  preparation  of  food,  remains  a relatively  “low-tech”  domain,  populated 
by  many  traditional,  manual,  or  mechanical  processes.  Thus,  as  a knowledge  area.  Food 
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Production  (Descriptor  #8)can  be  seen  as  rather  opposite  to  the  high-tech  areas  of  Computers  and 
Electronics  (Descriptor  #9).  For  this  reason,  this  factor  was  deemed  high  technology.  In  sum,  the 
factors  were  able  to  account  for  85%  of  the  variance  and  produced  clear  and  identifiable  factors 
to  describe  the  occupations  of  interest  herein. 

Occupation  Differences 

As  noted  above,  the  knowledges  were  designed  to  be  relatively  domain-specific. 
Therefore,  by  examining  the  descriptive  statistics  for  specific  occupations  (as  opposed  to  the 
averaged  results  in  Table  4-1),  we  should  begin  to  see  that  differentiation  becomes  apparent. 
Given  the  above,  the  results  presented  in  Table  4-12a  are  most  interesting.  Means  and  standard 
deviations  were  calculated  on  the  level  scale  for  six  occupations  selected  as  example^  these 
included:  (1)  General  Managers  and  Top  Executives,  (2)  Computer  Programmers,  (3)  Registered 
Nurses,  (4)  Police  Patrol  Officers,  (5)  Janitors  and  Cleaners,  and  (6)  Maintenance  Repairers, 
General  Utility.  The  results  support  the  specificity  of  the  knowledge  scales.  For  example,  the 
knowledge  area  Administration  and  Management  (Descriptor  #1)  showed  an  expectedly  high 
mean  for  managers  (M  = 4.84)  and  a low  mean  for  Janitors  (M  — .88).  General  Managers’  jobs 
were  also  rated  higher  on  knowledge  requirements  such  as  Personnel  and  Human  Resources 
(Descriptor  #6)  (M  = 4.52),  Customer  and  Personal  Service  (Descriptor  #5)  (M  = 4.66),  and 
Economics  and  Accounting  (Descriptor  #3)  (M  = 3.94).  On  the  other  hand.  General  Managers’ 
jobs  were  rated  low  on  the  level  of  knowledge  required  for  History  and  Archeology  (Descriptor 
#27)  (M  = 1.00)  and  Fine  Arts  (Descriptor  #26)  (M  = .50),  both  knowledge  areas  that  most 
managers  would  be  unlikely  to  need. 

, Computer  Programmers  rated  high  in  the  knowledge  requirements  one  might  expect,  with 
Computers  and  Electronics  (Descriptor  #9)  receiving  the  highest  rating  (M  = 6.56),  with  no  other 
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job  greater  than  3.2.  The  Mathematics  (Descriptor  #14)  requirement  rating  was  also  highest  (M  = 
4.67)  for  this  job.  Computers  and  Electronics  (Descriptor  #9)  knowledge  requirements  were  rated 
low  for  Maintenance  Repairers  (M  = 1.61)  and  Janitors  (M  = 1.08).  Nurses  showed  an  expected 
pattern,  with  incumbents  in  these  positions  rating  the  knowledge  requirements  of  Medicine  and 
Dentistry  (Descriptor  #21)  (M  = 4.91),  Psychology  (M  = 5.77),  and  Therapy  and  Counseling 
(Descriptor  #22)  (M  = 4.50)  highest.  Interestingly,  Police  Officers  also  rated  the  requirement  for 
Psychology  (Descriptor  #1 8)  very  high  (M  = 4.68)  along  with  English  Language  (Descriptor  #24) 
(M  = 4.05),  and,  as  expected,  very  high  ratings  for  the  levels  of  Public  Safety  and  Security 
(Descriptor  #29)  (M  = 5.79)  and  Law,  Government,  and  Jurisprudence  (Descriptor  #30)  (M  = 
5.00).  These  ratings  reflect  the  primary  activities  of  Police  Officers  in  enforcing  the  law,  and  in 
mediating  disputes  and  handling  difficult,  tense,  and  dangerous  situations. 

The  Janitors’  ratings  were  characterized  by  low  ratings  on  almost  all  the  knowledges 
including  Design  (Descriptor  #1 1)  (M  = 1.77)  and  Mechanical  (Descriptor  #13)  (M  = 1.69), 
although  the  standard  deviations  were  generally  a bit  higher  than  for  the  other  example 
occupations.  Maintenance  Repairers  rated  the  apparently  most  relevant  knowledge.  Mechanical 
(Descriptor  #13)  (M  = 4.3 1),  highest,  along  with  the  highest,  but  still  moderate,  ratings  for 
Building  and  Construction  (Descriptor  #12)  (M  = 3.86),  Design  (Descriptor  #1 1)  (M  = 2.72),  and 
Engineering  and  Technology  (M  = 2.80).  As  with  the  overall  ratings  shown  in  Table  4-1,  certain 
knowledges  were  rated  very  low  by  all  six  example  jobs  reflecting  the  relative  rarity  of  these 
knowledge  requirements  among  the  sample  occupations  studied.  These  knowledges  included 
areas  such  as  Food  Production  (Descriptor  #8),  Fine  Arts  (Descriptor  #26),  and  History  and 
Archeology  (Descriptor  #27). 
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Table  4-  12b  shows  the  ratings  obtained  from  incumbents  using  the  importance  scale 
applied  to  33  knowledges  for  the  same  six  jobs.  The  results  here  generally  support  those  obtained 
from  the  level  scale.  This  is  to  be  expected  given  that  the  two  scales  were  found  to  correlate  at 
.90  across  descriptors  within  jobs,  and  .95  across  jobs  within  descriptors  (see  Table  4-8).  Again, 
each  occupation  rated  the  same  knowledges  rated  highest  on  level  as  most  important.  For  ' 
example.  Managers  rated  Administration  and  Management  (Descriptor  #1)  (M  = 4.04),  Computer 
Programmers  rated  Computers  and  Electronics  (Descriptor  #9)  (M  = 4.67),  Police  rated  Public 
Safety  and  Security  (M  = 4.53),  and  Maintenance  Repairers  judged  Mechanical  (Descriptor  #13) 
(M  = 3.3 1)  as  most  important,  respectively. 

Discriminant  Analyses 

Having  examined  the  descriptive  statistics  for  the  knowledges  and  a sample  of 
occupations,  we  ran  a series  of  multivariate  analyses  to  better  understand  the  overall  pattern  of 
relationships  of  the  knowledge  measures.  Table  4-13  shows  the  results  of  the  first  of  these:  a 
discriminant  function  analysis.  Here,  the  knowledges  were  used  to  discriminate  among  the 
occupations  to  see  which  knowledge  areas  were  useful  in  differentiation.  While  most 
discriminant  function  analyses  produce  functions  with  multiple  indicators,  the  present  effort 
produced  an  unusual,  though  not  surprising,  result.  As  intimated  by  the  above  results,  it  was 
again  found  that  the  knowledges  were  successful  in  finding  discrete  differences  between  the  33 
occupations  in  the  sample.  In  fact,  the  results  showed  that  the  first  11  significant  functions  were 
marked  by  only  one  knowledge  each.  For  example,  the  first  function  was  characterized  by  only 
Clerical  (Descriptor  #2)  knowledge  (r  = .93),  the  second  by  only  Sales  and  Marketing  (Descriptor 
#4)  (r  = .84),  the  third  by  only  Public  Safety  and  Security  (Descriptor  #29)  (r  = .84),  and  so  on.  In 
fact,  among  the  first  1 1 functions,  the  next  highest  loading  besides  the  primary  one  (where  the 
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average  loading  was  .80)  for  each  function  was  only  .23.  Accordingly,  the  functions  were  named 
for  their  unique  identifying  knowledge.  This  unique  finding  lead  to  correct  classification  into 
their  occupations  of  55%  of  the  incumbents.  The  conclusion  is  that  the  knowledges  seemed  to  be 
acting  as  designed;  as  unique,  relatively  domain-specific  markers  of  one  aspect  of  occupation 
description. 

Convergence  With  Analysts’  Ratings 

As  noted  above,  we  were  able  to  obtain  ratings  on  the  knowledges  from  both  job 
incumbents  and  occupational  analysts.  This  parallel  data  collection  allowed  for  comparisons 
between  the  incumbent  and  occupational  analyst  responses.  Overall,  the  results  of  these  analyses 
allow  us  to  come  to  similar  conclusions  about  the  occupations  represented  and  the  knowledge 
requirements.  However,  there  are  several  differences  and  unique  findings  that  are  worth 
reporting.  Most  importantly,  the  analysts  and  the  incumbents  were  generally  in  agreement  with 
regards  to  their  ratings  assigned  to  the  abilities.  The  average  correlation  across  knowledges 
between  the  groups  was  .65  for  both  level  and  importance.  This  indicates  that  the  two  groups 
were  in  general  agreement  on  the  level  and  importance  of  the  various  knowledges  across 
occupations. 

As  for  specifics.  Tables  14a  and  14b  shows  a summary  of  the  means,  standard 
deviations,  reliabilities,  and  several  comparisons  between  the  two  set  of  respondent’s  ratings  for 
the  level  and  importance  scale.  There  are  several  findings  that  emerge  when  examining  these 
tables.  First,  for  the  level  scale,  the  analysts  tended  to  provide  more  reliable  ratings  (Manaiysts  = 

.86  vs.  Mincum.  = .76)  despite  the  fact  that  there  were  fewer  of  them  (Manaiysts  = 9.81  vs.  Mincum.  = 
19.66)  rating  the  target  occupations  relative  to  the  number  of  incumbents.  Nonetheless,  both 
analysts’  and  incumbents’  reliabilities  were  sufficiently  high  for  the  intended  purposes. 
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The  second  finding  is  that  there  were  a number  of  significant  differences  in  both  the  mean 
ratings  and  the  variance  of  those  ratings  between  the  two  groups  of  raters.  Overall,  20  of  the  t- 
tests  showed  significant  differences  as  did  21  of  the  F variance  tests.  While  for  the  most  part  the 
incumbents  tended  to  rate  the  knowledges  higher  than  did  the  analysts,  there  were  several  scales, 
including  the  levels  of  Mechanical  (Descriptor  #13)  (t(971)  = -6.51)  and  Law,  Government,  and 
Jurisprudence  (Descriptor  #30)  ($(971)  = -5.11)  knowledge  required  where  the  analysts  provided 
higher  ratings.  In  general,  however,  the  incumbents’  ratings  tended  to  be  less  than  one-half  a 
standard  deviation  higher  than  the  analysts.  Furthermore,  the  d2  statistics  show  that  the  averaged, 
absolute  mean  score  difference  was  generally  about  one  point  on  the  seven-point  scale  (using  the 
square  root  of  the  mean  d2  as  the  indicator)  and  about  .5  to  .75  points  on  the  five-point 
importance  scale. 

We  ran  a principal  components  analysis  of  the  analyst  data,  using  the  same  assumptions 
and  rotation  that  had  been  used  with  the  incumbent  data.  It  was  hoped  that  the  comparison 
between  the  two  solutions,  based  on  analyst  and  incumbents,  respectively,  would  shed  some 
further  light  on  the  relationship  between  the  ratings  provided  by  these  two  groups.  The  results  of 
this  analysis  are  presented  in  Table  4-16,  and,  in  general,  the  two  groups’  factors  were  relatively 
similar  except  for  differences  in  the  order  in  which  the  factors  appeared.  The  first  analyst  factor 
was  termed  General  Management,  as  it  included  the  Business  Administration  knowledges  along 
with  several  others.  Specifically,  this  factor  was  marked  by  Personnel  and  HR  (r  = .88), 
Administration  and  Management  (Descriptor  #1)  (r  = .87),  and  Economics  and  Accounting 
(Descriptor  #3)  (r  = .84),  as  was  the  incumbent’s  factor.  However,  the  analysts’  factor  also  added 
English  Language  (Descriptor  #24)  (r  = .72)  and  Education  and  Training  (Descriptor  #23)  (r  = 

.69),  among  others,  making  it  a slightly  broader  representation  of  managerial  knowledges.  The 
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second  analyst  factor  was  a narrower  version  of  the  incumbent’s  Science  and  Technology,  here 
called  Engineering,  most  prominently  marked  by  Engineering  and  Technology  (Descriptor  #10) 

(r  = .91),  Physics  (Descriptor  #15)  (r  s .91),  and  Design  (Descriptor  #1 1)  (r  = .84).  Analyst  factor 
three  continued  the  “not  quite  the  same”  trend  with  a broader  version  of  the  incumbent’s 
Medicine  factor.  This  grouping,  called  Allied  Health  Services  included  the  Medicine  and 
Dentistry  (Descriptor  #21)  (r  = .95),  Biology  (Descriptor  #17)  (r  = .86),  and  Chemistiy 
(Descriptor  #16)  (r  = .66),  but  also  added  the  Therapy  and  Counseling  (Descriptor  #22)  (r  = .80) 
and  Psychology  (Descriptor  #18)  (r  = .52)  knowledge  areas. 

The  next  factor  was  very  similar  to  the  Law  Enforcement  incumbent  factor  with  similar 
markers  such  as  Public  Safety  and  Security  (Descriptor  #29)  (r  = .60)  and  Law,  Government,  and 
Jurisprudence  (Descriptor  #30)  (r  = .56),  but  it  too  was  slightly  broader  with  Geography 
(Descriptor  #20)  (r  = .82)  and  Customer  and  Personal  Service  (Descriptor  #5)  (r  = .42)  loadings, 
along  with  no  fewer  than  seven  other  knowledges  with  loadings  above  .30  (the  cutoff  for 
inclusion).  Factor  six,  High  Technology,  was  similar  as  well  to  an  incumbent  factor  with  high 
positive  loadings  on  Computers  and  Electronics  (Descriptor  #9)  (r  = .72),  Telecommunications 
(Descriptor  #31)  (r  = .65),  and  negatively  on  Food  Production  (Descriptor  #8)  (r  = -.80).  Once 
again,  there  were  a number  of  other  knowledges  loading  highly  including  Transportation 
(Descriptor  #33)  (r  = .80),  Clerical  (Descriptor  #2)  (r  = .57)  and  Communications  and  Media 
(Descriptor  #32)  (r  = .32).  The  final  analyst  factor  was  termed  Clerical  (Descriptor  #2)  and  was 
notable  for  high  positive  loadings  on  English  Language  (Descriptor  #24)  (r  = .93)  and  Clerical 
(Descriptor  #2)  (r  = .44)  and  high  negative  loadings  on  Building  and  Construction  (Descriptor 
#12)  (r  = -.56)  and  Mechanical  (Descriptor  #13)  (r  = -.42).  This  Clerical  (Descriptor  #2)  factor 
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was  therefore  described  by  both  the  presence  of  Clerical  (Descriptor  #2)  knowledge  areas  and  the 
extreme  lack  of  certain  outdoor  and/or  physical  knowledge  areas. 

The  above  results  suggest  that,  although  the  analysts  seemed  to  provide  more  reliable 
ratings,  their  responses  were  a bit  less  cogent  in  terms  of  the  relationships  among  the  knowledges 
for  the  given  set  of  occupations.  For  example,  many  of  the  analyst  factors  were  less  “clean”  than 
were  the  incumbents’,  with  sometimes  extraneous  or  only  tangentially  related  knowledges 
appearing  on  a factor.  Further , the  analysts’  factor  solution  showed  more  loadings  below  .50  (34 
for  analysts  vs.  26  for  incumbents)  and  more  negative  loadings  (10  vs.  4),  suggesting  that  the 
analysts  had  a less  clear  picture  of  how  the  knowledges  were  related  to  each  other  within  the  ' 
given  set  of  occupations.  While  the  above  is  only  a qualitative  assessment,  it  does  provide  some 
evidence  for  both  the  general  similarity  of  the  incumbent  and  analysts  ratings  (the  similar  means 
and  factor  structures).  At  the  same  time  there  is  some  evidence  that  these  two  groups  differed 
somewhat  in  their  specific  perceptions  of  the  knowledge  requirements  for  the  rated  occupations. 

To  see  if  analyst  and  incumbent  ratings  differed  depending  on  the  scale  coding,  means, 
standard  deviations,  reliabilities,  and  t-tests  were  calculated  for  each  ability  using  the  relevant/not 
relevant  (zero-one)  coding  scheme.  The  results  of  this  analysis  are  presented  in  Table  4-15  and 
show  that  analyst  and  incumbent  ratings  are  fairly  similar,  even  under  this  rescoring  scheme. 

Conclusions 

The  results  reviewed  here  suggest  that  the  proposed  taxonomy  of  job-related  knowledges 
is  a useful  descriptive  and  interpretative  tool  for  trying  to  understand  and  measure  the  types  and 
levels  of  various  knowledges  required  in  a wide  variety  of  occupations.  The  Knowledge 
Requirements  Taxonomy  and  measurement  system  described  herein  is  based  on  an  extension  of 
the  ability  requirements  approach  developed  by  Fleishman  and  his  colleagues  (see  Fleishman  & 
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Quaintance,  1984;  Fleishman,  1975,  1991;  Fleishman  & Mumford,  1991).  This  methodology  has 
been  used  to  develop  constructs  and  associated  measurement  scales  exhibiting  high  reliability, 
internal  validity,  and  external  validity.  The  knowledge  scales  were  created  beginning  with  a 
review  of  the  cognitive,  vocational,  training,  and  job  analysis  literatures.  Knowledge  categories 
were  broadened,  narrowed,  altered,  or  discarded  based  on  the  review,  ratings,  and  comments  of 
multiple  professional  psychologists.  Task  anchored  measurement  scales  were  also  developed 
empirically  so  that  the  task  anchors  represented  different  levels  of  a particular  knowledge  and 
had  high  reliability  with  regard  to  their  positions  on  the  scales.  Special  attention  was  given  to 
making  the  scales  readable,  understandable,  and  "user  friendly." 

The  present  study  confirms  that  the  knowledge  scales  developed  have  high  reliabilities 
when  used  with  job  incumbents  or  with  occupational  analysts  and  further  establishes  their  utility 
in  describing  and  understanding  worker  performance  for  multiple  jobs.  The  scales  also  appear 
useful  in  meaningfully  classifying  occupations  in  terms  of  the  underlying  knowledges  needed  to 
perform  them.  As  part  of  the  Department  of  Labor's  new  Occupational  Information  System,  the 
knowledge  taxonomy  and  measurement  system  should  make  an  important  contribution  in  the 
understanding  of  worker  characteristics  required  to  successfully  perform  a very  wide  variety  of 
job  tasks. 

The  inclusion  of  occupational  specialty  data  with  each  rating  scale  in  this  study  should 
allow  linkage  of  the  knowledges  to  other  national  occupational  and  educational  databases.  The 
present  study  showed  how  such  linkages  could  be  developed  with  the  National  Occupational 
Information  Coordinating  Committee  (NOICC)  clustering  structure.  This  linkage  provides  a 
mechanism  for  matching  job  market  demand  and  institutional  supply  data  gathered  by  State 
Occupational  Information  Coordinating  Committees  (SOICC)  with  knowledge  requirements. 
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Future  work  should  be  directed  at  examining  these  linkages  more  closely.  The  positive  results 
obtained  thus  far  are  encouraging  for  further  applications  of  these  scales  to  define  the  knowledge 
requirements  of  a wide  variety  of  occupations. 

Some  specific  methodological  results  have  implications  for  future  development  and 
application.  The  reliabilities  of  the  scales  with  a relatively  limited  sample  of  occupations  and 
number  of  incumbents  within  occupation  indicates  that  even  higher  reliabilities  would  result  with 
larger  samples.  Also,  the  high  correlations  obtained  between  “level  of  knowledge”  and 
“importance  of  the  knowledge”  scales  across  occupations  indicates  that  one  scale  might  suffice. 
The  “level”  scale  appears  to  be  more  appropriate  for  most  anticipated  uses.  The  knowledges  in 
the  taxonomy  were  effective  in  describing  a wide  variety  of  occupations.  Thus,  mean  ratings  of 
knowledge  requirements,  on  the  whole,  were  low  across  many  occupations,  but  high  for  specific 
occupations.  As  a set,  the  knowledge  scales  were  shown  to  cover  the  requirements  for  the  whole 
range  of  occupations,  from  top  management  to  janitors.  Also,  the  multiple  revision  process 
followed  captured  a broad  range  of  knowledges  at  a relatively  similar  level  of  specificity. 

The  knowledge  content,  structure,  and  scales  should  prove  useful  in  areas  such  as  job 
analysis,  person/job  matching,  job  training  and  retraining,  career/occupational  counseling, 
vocational  interest  assessment,  and  the  development  of  job  families.  Used  in  concert  with 
existing  taxonomies  of  work-related  abilities,  skills,  and  competencies,  the  knowledge  scales  will 
help  us  to  more  completely  and  accurately  describe  and  understand  the  world  at  work. 
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Figure  4-1 
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Figure  4-1  (continued) 
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Figure  4-1  (continued) 
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Figure  4-1  (continued) 
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Figure  4-1  (continued) 
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Figure  4-1  (continued) 
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Figure  4-1  (continued) 

Descriptions  and  Definitions  of  Knowledges 
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Figure  4-2 

Example  Page  from  Knowledges  Questionnaire 


Knowledge  of  planning,  coordination,  and  execution  of 
business  functions,  resource  allocation,  and  production. 


1.  Administration 
and  Management 

Level 

What  level  of  this  knowledge  is  needed  to  perform  this  job? 
7 


Requires  knowledge  of  high-level 
business  administration  such  as  being  the 
CEO  of  a major  industrial  company. 


Requires,  knowledge  of  basic  management 
such  as  monitoring  a group  filling  out  job 
applications. 


Manage  a $10  million  company. 


Administer  a large  retirement  and 
nursing  care  facility. 

Monitor  progress  of  a project  to 
ensure  timely  completion. 


Plan  an  effective  staff  meeting. 
Sign  a pay  voucher. 


NR  Not  relevant  at  all  for  performance  on  this  job. 


Not 

Important 
- — I 


Importance 

How  important  is  this  knowledge  to  performance  on  this  job? 

Somewhat  T Very 

Important 


Important 


Important 


■4- 


Extremely 

Important 


1 1 r 

1 2 3 4 5 

Job  Specialty  Requirements 

Which  of  the  following  specialties  are  relevant  to  this  job?  (Mark  “R”  for  Relevant  and  “NR”  for  Not  Relevant.) 


R NR  Business  Administration 

R NR  Medical  Service  Management 

R NR  Construction  Management 

R NR  Personnel  and  Human  Resource  Management 

R NR  Engineering,  Mathematical,  and  Sciences 

R NR  Public  Administration 

Management 

R NR  Food  Service  and  Lodging  Management 

Otherfs) 

(Please  specify) 

313 


Figure  4-3 

Thirty-Three  Occupations  With  Four  or  More  Incumbents  Completing  the  Knowledges 
Questionnaire 


Occupation 

Code 

Occupation  Title 

Number  of 
Respondents 

15005 

Education  Administrators 

8 

19005 

General  Managers  & Top  Executives 

50 

22114 

Chemical  Engineers 

4 I 

25105 

Computer  Programmers 

9 

27311 

Recreation  Workers 

6 

31303 

Teachers,  Preschool 

5 

31305 

Teachers,  Elementary  School 

7 

32502 

Registered  Nurses 

22 

32902 

Medical  & Clinical  Laboratory  Technologists 

4 

49008 

Salespersons,  Except  Scientific  & Retail 

ii 

49011 

Salespersons,  Retail 

20 

49021 

Stock  Clerks,  Sales  Floor 

9 

49023 

Cashiers 

23 

51002 

First  Line  Supervisors,  Clerical/Administrative 

58 

53905 

Teachers’  Aides  & Assistants,  Clerical 

4 

55108 

Secretaries 

83 

55305 

Receptionists  & Information  Clerks 

13 

55338 

Bookkeeping,  Accounting,  & Auditing  Clerks 

28 

55347 

General  Office  Clerks 

83 

61005 

Police  & Detective  Supervisors 

12 

63014 

Police  Patrol  Officers 

19 

65008 

Waiters  & Waitresses 

14 

65026 

Cooks,  Restaurant 

6 

65038 

Food  Preparation  Workers 

32 

66008 

Nursing  Aides,  Orderlies,  & Attendants 

18 

67005 

Janitors  & Cleaners 

26 

85119 

Other  Machinery  Maintenance  Mechs 

4 

85132 

Maintenance  Repairers,  General  Utility 

36 

87902 

Earth  Drillers,  Except  Oil  & Gas 

5 

92974 

Packaging  & Filling  Machine  Operators 

10  | 

97102 

Truck  Drivers 

7 

97111 

Bus  Drivers,  Schools 

9 1 

Table  4-1 


Between  Occupations:  Kn 
Descriptor 

owledees 

Variable 

Level 

Importance 

M 

SD 

SEMa 

Ikb 

M 

SD 

SEM 

fie 

1 . Admin,  and  Mgmt. 

2.35 

2.16 

.76 

.88 

2.51 

1.45 

.50 

.87 

2.  Clerical 

3.30 

2.55 

.57 

.95 

2.83 

1.45 

.37 

.93 

3.  Economics  and  Acct. 

1.82 

2.21 

.87 

.84 

2.09 

1.33 

.54 

.83 

4.  Sales  and  Marketing 

1.41 

2.10 

.75 

.87 

1.81 

1.22 

.42 

.87 

5.  Cust.  and  Pers.  Svc 

2.75 

2.48 

1.11 

.80 

2.63 

1.47 

,71 

.76 

6.  Personnel  and  HR 

2.17 

2.25 

.95 

.82 

2.27 

1.36 

.61 

.79 

7.  Production  and  Proc. 

1.21 

2.08 

1.13 

.70 

1.64 

1.13 

.60 

.71 

8.  Food  Production 

.68 

1.70 

1.15 

.54 

1.35 

.88 

.49 

.68 

9.  Computers  and  Elect. 

2.43 

2.06 

.85 

.83 

2.50 

1.36 

.53 

.84 

10.  Engineering  and  Tech. 

1.10 

2.00 

.98 

.76 

1.57 

1.07 

.49 

.78 

1 1 . Design 

1.14 

2.03 

1.22 

.63 

1.56 

1.06 

.66 

.60 

12.  Building  and  Constr. 

1.06 

2.09 

.90 

.81 

1.53 

1.08 

.43 

.84 

13.  Mechanical 

1.21 

2.04 

,79 

.85 

1.60 

1.06 

.73 

.87 

14.  Mathematics 

2.93 

1.82 

.76 

.82 

2.82 

1.14 

.51 

.79 

15.  Physics 

.91 

1.82 

.91 

.75 

1.46 

.95 

.44 

.78 

16.  Chemistry 

1.21 

2.00 

.82 

.83 

1.66 

1.13 

.46 

.83 

17.  Biology 

.87 

1.92 

.84 

.81 

1:44 

1.02 

.41 

.83 

18.  Psychology 

2.49 

2.38 

.84 

.88 

2.35 

1.36 

.51 

.85 

19.  Sociology  and  Anthro. 

1.22 

1.89 

.79 

.82 

1.60 

1.01 

.42 

.82 

20.  Geography 

1.31 

1.98 

1.22 

.62 

1.65 

1.04 

.62 

.64 

21.  Medicine  and  Dent. 

1.17 

2.02 

.86 

.82 

1.73 

1.27 

.46 

.86 

22.  Therapy  and  Couns. 

1.39 

2.13 

.82 

.85 

1.77 

1.20 

.46 

.84 

23.  Education  and  Tm. 

2.30 

2.31 

1.00 

.81 

2.23 

1.28 

.55 

.81 

24.  English  Language 

3.18 

2.01 

.78 

.85 

2.98 

1.23 

.53 

.81 

25.  Foreign  Language 

.86 

1.71 

1.36 

.37 

1.41 

.84 

.63 

.43 

26.  Fine  Arts 

.58 

1.52 

1.04 

.53 

1.28 

.75 

.45 

.63 

27.  History  and  Archeol. 

.71 

1.61 

.92 

.67 

1.35 

.86 

.45 

.72 

28.  Philosophy  and  The. 

.99 

1.75 

.87 

.75 

1.49 

.93 

.45 

.76 

29.  Public  Safety  and  Sec. 

1.71 

2.11 

.76 

.87 

2.10 

1.35 

.54 

.83 

30.  Law,  Govt.,  and  Jurisp. 

1.54 

2.05 

.82 

.84 

1.90 

1.24 

.49 

.84 

3 1 . Telecommunications 

1.42 

1.73 

1.25 

.47 

2.08 

1.15 

.77 

.55 

32.  Comm  and  Media 

1.72 

1.88 

1.06 

.68 

2.00 

1.13 

.70 

.61 

33.  Transportation 

1.02 

1.67 

1.00 

.64 

1.57 

.97 

.57 

.65 

— Statistics  are  based  on  33  occupations  with  Knowledges  questionnaire  responses  from  at 
least  4 incumbents  (mean  number  of  incumbents  = 19.66,  median  = 1 1.00,  harmonic  mean  = 
9.18). 


Table  4-1  (continued) 

Descriptive  Statistics  across  All  Occupations  and  Reliability  Estimates  for  Rated  Differences 
Between  Occupations:  Knowledges 

aThis  estimate  of  the  standard  error  of  measurement  was  calculated  as  SEM  = SD*  ^/(l- rt) 

bThis  estimate  of  reliability  was  obtained  by  calculating  the  intraclass  correlation  for  k ratings 
across  occupations:  ICC(l,k)  = fBMS-WMSI/BMS  (Shrout  & Fleiss,  1979),  where  k is  the 
harmonic  mean  of  the  number  of  ratings  provided  on  each  occupation. 


Table  4-2 

Reliability  of  Rated  Differences  Between 


Occupations  Considering  Varying  Numbers  of  Raters: 


Knowledges 


Variable 


Level 


Descriptor 

II* 

_ b 
130 

ri 

130 

1.  Admin  and  Mgmt. 

.39 

.95 

.39 

.95 

2.  Clerical 

.62 

.98 

.56 

.97 

3.  Economics  and  Acct. 

.32 

.94 

.31 

.93 

4.  Sales  and  marketing 

.37 

.95 

.39 

.95 

5.  Cust.  and  Pers.  Svc 

.26 

.91 

.22 

.90 

6.  Personnel  and  HR 

.29 

.92 

.26 

.91 

7.  Production  and  Proc. 

.17 

.86 

.18 

.87 

8.  Food  Production 

.09 

.76 

.16 

.85 

9.  Computers  and  Elect. 

.30 

.93 

.33 

.94 

10.  Engineering  and  Tech. 

.22 

.89 

.25 

.91 

11.  Design 

.13 

.82 

.12 

.80 

12.  Building  and  Constr. 

.28 

.92 

.31 

.93 

13.  Mechanical 

.33 

.94 

.38 

.95 

14.  Mathematics 

.29 

.93 

.26 

.91 

15.  Physics 

.21 

.89 

.25 

.91 

16.  Chemistry 

.31 

.93 

.31 

.93 

17.  Biology 

.27 

.92 

.31 

.93 

18.  Psychology 

.39 

.95 

.35 

.94 

19.  Sociology  and  Anthro. 

.29 

.93 

.30 

.93 

20.  Geography 

.13 

.81 

.14 

.83 

21.  Medicine  and  Dent. 

.28 

.92 

.36 

.94 

22.  Therapy  and  Couns. 

.33 

.94 

.33 

.94 

23.  Education  and  Tm. 

.28 

.92 

.28 

.92 

24.  English  Language 

.33 

.94 

.28 

.92 

25.  Foreign  Language 

.05 

.61 

.06 

.67 

26.  Fine  Arts 

.09 

.75 

.14 

.82 

27.  History  and  Archeol. 

.15 

.84 

.19 

.87 

28.  Philosophy  and  The. 

.21 

.89 

.23 

.90 

29.  Public  Safety  and  Sec. 

.37 

.95 

.31 

.93 

30.  Law,  Govt.,  and  Jurisp. 

.32 

.93 

.32 

.93 

31.  T elecommunications 

.07 

.70 

.10 

.76 

32.  Comm,  and  Media 

.16 

.85 

.12 

.80 

33.  Transportation 

.14 

.82 

.14 

.83 

Note.  Reliability  estimates  are  based  on  a 33  occupations  with  Knowledges  questionnaire 
responses  from  at  least  4 incumbents  (mean  number  of  incumbents  = 19.66,  median  =11.00, 
harmonic  mean  = 9.18).  Decimals  are  omitted. 


Table  4-2  (continued) 

Reliability  of  Rated  Differences  Between  Occupations  Considering  Varying  Numbers  of  Raters: 
Knowledges 

aSingle  rater  estimates  of  reliability  were  obtained  by  calculating  the  intraclass  correlation  for 
single  judges  ratings  across  occupations:  KX(1,1)  = fBMS-WMSl/IBMS+fk+lYWMSl  (Shrout 
& Reiss,  1979),  where  k is  the  harmonic  mean  of  the  number  of  ratings  provided  on  each 
occupation. 

Estimates  of  reliability  for  30  raters  were  obtained  by  applying  the  Spearman-Brown  correction 
formula  to  the  single  rater  reliability  estimates. 


V 


Table  4-3 

Reliability  of  Rated  Differences 


Between  Occupations  Considering  Various  Rernri in p Sdien^s: 


Knowledges 


Type  of  Scale  and  Recoding  Applied 
Level 


Descriptor 

la 

rb 

la 

Tb 

1.  Admin,  and  Mgmt. 

88 

83 

83 

87 

92 

2.  Clerical 

95 

85 

89 

93 

96 

3.  Economics  and  Acct. 

84 

77 

83 

83 

84 

4.  Sales  and  Marketing 

87 

88 

83 

87 

84 

5.  Cust.  and  Pers.  Svc. 

80 

83 

67 

76 

83 

6.  Personnel  and  HR 

82 

84 

78 

79 

85 

7.  Production  and  Proc. 

70 

87 

64 

71 

79 

8.  Food  Production 

54 

81 

71 

68 

62 

9.  Computers  and  Elect. 

83 

86 

81 

84 

84 

10.  Engineering  and  Tech. 

76 

68 

76 

78 

84 

11.  Design 

63 

75 

66 

60 

76 

12.  Building  and  Constr. 

81 

81 

82 

83 

92 

13.  Mechanical 

85 

85 

80 

87 

93 

14.  Mathematics 

82 

60 

79 

79 

81 

15.  Physics 

75 

78 

80 

78 

84 

16.  Chemistry 

83 

72 

79 

83 

89 

17.  Biology 

81 

46 

82 

83 

85 

18.  Psychology 

88 

71 

69 

85 

91 

19.  Sociology  and  An  thro. 

82 

72 

80 

82 

89 

20.  Geography 

62 

67 

69 

63 

44 

2 1 . Medicine  and  Dent. 

82 

71 

80 

86 

92 

22.  Therapy  and  Couns. 

85 

82 

84 

84 

89 

23.  Education  and  Tm. 

81 

80 

72 

81 

83 

24.  English  Language 

85 

62 

80 

81 

84 

25.  Foreign  Language 

37 

85 

57 

43 

48 

26.  Fine  Arts 

53 

85 

63 

63 

41 

27.  History  and  Archeol. 

67 

87 

67 

71 

66 

28.  Philosophy  and  The. 

75 

91 

76 

76 

87 

29.  Public  Safety  and  Sec. 

87 

88 

70 

83 

82 

30.  Law,  Govt.,  and  Jurisp. 

84 

75 

82 

84 

82 

31.  Telecommunications 

47 

84 

60 

54 

52 

32.  Comm,  and  Media 

68 

91 

68 

60 

67 

33.  Transportation 

64 

90 

61 

64 

57 

Reliability  estimates  are  based  on  33  occupations  with  Knowledge  questionnaire  responses 
from  at  least  4 incumbents  (mean  number  of  incumbents  = 19.66,  median  = 1 1.00,  harmonic 
mean  = 9.18).  Reliability  estimates  stipulated  as  £ were  calculated  using  the  full  eight  point  scale 
for  level,  and  retaining  all  of  the  data  for  the  importance  scale.  Reliability  estimates  stipulated  as 
rb  were  calculated  using  a reduced  seven  point  scale  for  level,  and  excluding  the  data  for 
importance  scale  where  the  rater  marked  “NR”  on  the  level  scale.  Reliability  estimates  stipulated 


Table  4-3  (continued) 

Reliability  of  Rated  Differences  Between  Occupations  Considering  Various  Recoding  Schemes: 
Knowledges 


as  rc  were  calculated  using  binary  coded  scale  for  level  (relevant/not  relevant).  Decimals  are 
omitted. 


Table  4-4a 


i-saiaiyoio  ui  v aiicuiu^  iui 

Variation  on  the  Level  Scale:  Knowledges 
Source  of  Variation 

iuun,  ana 

ss 

levani  ini 
df 

teracuons  i 
MS 

is  sources  or 
F 

Occupations 

6204.22 

32 

193.88 

3.79* 

S(Occupations) 

31547.65 

616 

51.21 

Knowledge 

3898.36 

32 

121.82 

66.50* 

Knowledge  x Occupations 

12951.47 

1024 

12.65 

6.90* 

Knowledge  x S (Occupations) 

36113.58 

19712 

1.83 

Note.  Occupations  are  treated  as  random,  between-subjects  effects,  while  descriptors  are 
treated  as  fixed,  within  subjects  effects. 

*P<.05 


Table  4-4b 

Analysis  of  Variance  for  Descriptor,  Occupation,  and  Relevant  Interactions  as  Snnrras  nf 


Source  of  Variation 

SS 

df 

MS 

F 

Occupations 

3411.72 

32 

106.62 

3.46* 

S(Occupations) 

18969.68 

616 

30.79 

Knowledge 

3610.51 

32 

112.83 

70.91* 

Knowledge  x Occupations 

10980.07 

1024 

10.72 

6.74* 

Knowledge  x S(Occupations) 

31365.54 

19712 

1.59 

Note.  Occupations  are  treated  as  random,  between-subjects  effects,  while  descriptors  are 
treated  as  fixed,  within  subjects  effects. 

*p<.05 


Table  4-5 

Interrater  Agreement  Coefficients  for  Each  Scale  Type:  Knowledges 


Number  of  Raters  on  Each  Variable 

Scale  Type 

Ik 

Ii 

130 

Level 

.86 

.40 

.95 

Importance 

.85 

.39 

.94 

Note.  Interater  agreement  coefficient  estimates  are  based  on  33  occupations  with  Knowledges 
questionnaire  responses  from  at  least  4 incumbents  (mean  number  of  incumbents  = 19.66, 
median  = 1 1.00,  harmonic  mean  = 9.18).  Full  sample  interrater  agreement  coefficients  (rk)  were 
obtained  by  considering  the  “Descriptor  x Occupations”  terms  from  tables  4-4a  and  4-4b  as  true 
variance.  Error  variance  was  defined  as  the  “Descriptor  x S (Occupations)”  term.  Estimated  for 
reliability  for  1 and  30  raters  were  obtained  by  applying  the  Spearman-Brown  correction  formula 
to  the  k rater  reliability  estimates,  where  k is  the  harmonic  mean  of  the  number  of  raters  for  each 
occupation. 
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Table  4-6a 

Analysis  of  Variance  for  Aggregate  Descriptor,  Occupation,  and  Relevant  Interactions  as 
Sources  of  Variation  on  the  Level  Scale:  Knowledges 


Source  of  Variation 

SS 

df 

MS 

F 

Occupations 

1976.30 

32 

61.76 

3.74* 

S (Occupations) 

10159.97 

616 

16.49 

Aggregate 

532.55 

9 

59.17 

59.97* 

Aggregate  x Occupations 

1937.30 

288 

6.73 

6.82* 

Aggregate  x S(Occupations) 

5470.09 

5544 

.99 

Note.  Occupations  are  treated  as  random,  between-subjects  effects,  while  descriptors  are 
treated  as  fixed,  within  subjects  effects. 

*P<.05 
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Table  4-6b 

Malysis  of  Variance  for  Aggregate  Descriptor.  Occupation,  and  Relevant  Interactions  as  Sources 
of  Variation  on  the  Importance  Scale:  Knowledges 


Source  of  Variation 

SS 

df 

MS 

F 

Occupations 

1141.40 

32 

35.67 

3.53* 

S(Occupations) 

6221.37 

616 

10.10 

Aggregate 

428.86 

9 

47.65 

54.28* 

Aggregate  x Occupations 

1731.58 

288 

6.01 

6.85* 

Aggregate  x S (Occupations) 

4866.65 

5544 

.88 

Note,  Occupations  are  treated  as  random,  between-subjects  effects,  while  descriptors  are  treated 
as  fixed,  within  subjects  effects. 

*p<.05 


Table  4-7 

Interrater  Agreement  Coefficients  for  Aggregate  Descriptors  for  Each  Scale  Type:  Knowledges 


Number  of  Raters  on  Each  Variable 

Scale  Type 

Dc 

Ii 

130 

Level 

.85 

.39 

.94 

Importance 

.85 

.39 

.94 

Note.  Interrater  agreement  coefficients  estimates  are  based  on  33  occupations  with  Knowledges 
questionnaire  responses  from  at  least  4 incumbents  (mean  number  of  incumbents  = 19.66, 
median  = 1 1.0,  harmonic  mean  = 9.18).  Full  sample  interrater  agreement  coefficients  (r^were 
obtained  by  considering  the  “Aggregate  x Occupations”  term  from  Tables  4-6a  and  4-6b  as  hue  - 
variance.  Error  variance  was  defined  as  the  “Aggregate  x S(Occupations)”  term.  Estimates  of 
reliability  for  1 and  30  raters  were  obtained  by  applying  the  Spearman-Brown  correction  formula 
to  the  k rater  reliability  esitmates,  where  k is  the  harmonic  mean  of  the  number  of  raters  for  each 
occupation. 
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Table  4-8 

Means  and  Standard  Deviations  of  Correlations  Between  the  Level  anti  Importance  Scales  Across 
Occupations  and  Descriptors:  Knowledges 

Level Importance 

Scale na  M SD n M SD 

Level  — — — 33  .90  .21 

Importance 33  .95  .04  

Note.  All  correlations  were  calculated  based  on  the  mean  of  ratings  assigned  by  raters  for  a given 
occupation,  descriptor,  and  scale.  Level-Importance  Means  above  the  diagonal  were  calculated 
by  taking  the  level  scale  means  on  a given  occupation  for  all  descriptors,  correlating  them  with 
importance  scale  means,  for  that  occupation,  and  then  averaging  them  with  the  correlations  for 
other  occupations.  Level-Importance  means  below  the  diagonal  were  calculated  by  taking  the 
level  scale  means  for  a given  descriptor  for  all  occupations,  correlating  them  with  importance 
scale  means,  for  that  descriptor,  and  averaging  them  with  correlations  for  other  descriptors.  Other 
means  in  the  table  were  calculated  in  a similar  manner. 

aNumber  of  correlations  averaged,  not  number  of  observations  on  which  correlations  were 
calculated. 
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Table  4-9b  . ■ 

Intercorrelations  of  Descriptors  for  the  Importance  Scale  (Occupation-Level  Data):  Knowledges 
Descriptor  1 2 3456789  10  11 
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Table  4-11 

Principal  Components  Analysis  Pattern  Matrix  for  the  Level  Scale:  Knowledges  (loadings  over 


Factor 


Descriptor 

FI 

F2 

F3 

F4 

F5 

F6 

F7 

Communality 

1.  Admin,  and  Mgmt. 

.21 

.34 

.40 

.56 

.05 

.50 

.02 

.88 

2.  Clerical 

.06 

-.16 

.02 

.82 

-.04 

.04 

.19 

.73 

3.  Economics  and  Acct. 

.09 

.30 

.14 

.50 

-.25 

.58 

-.01 

.76 

4.  Sales  and  Marketing 

.02 

.02 

-.05 

-.00 

.02 

.95 

.07 

.91 

• 5.  Cust.  & Pers.  Svc. 

.22 

-.39 

.12 

.18 

.23 

.71 

-.00 

.80 

6.  Personnel  and  HR 

.14 

.12 

.46 

.51 

.14 

.55 

-.20 

.87 

7.  Production  Proc. 

.02 

.75 

.04 

.14 

.02 

.12 

-.28 

.67 

8.  Food  Production 

.36 

.27 

-.09 

-.14 

-.09 

-.00 

-.78 

.84 

9.  Computers  and  Elect. 

.02 

.26 

.06 

.56 

.04 

.03 

.66 

.82 

10.  Engineering  and  Tech. 

-.11 

.92 

.10 

.11 

.13 

-.16 

.13 

.94 

11.  Design 

.14 

.89 

.10 

.10 

-.17 

.03 

.08 

.87 

12.  Building  and  Constr. 

.19 

.92 

.03 

-.13 

-.06 

.12 

-.04 

.91 

13.  Mechanical 

.03 

.94 

.04 

-.09 

.18 

-.01 

-.06 

.92  ‘ 

14.  Mathematics 

.09 

.53 

.02 

.46 

.28 

.29 

.46 

.87 

15.  Physics 

.33 

.74 

.05 

.08 

.43 

-.20 

.04 

.89 

16.  Chemistry 

.01 

.64 

.12 

-.01 

.72 

-.05 

.14 

.96 

17.  Biology 

.19 

-.04 

.02 

.09 

.95 

.00 

.02 

.94 

18.  Psychology 

.41 

-.02 

.68 

.21 

.29 

.17 

.21 

.83 

19.  Sociology  and  Anthro 

.68 

-.01 

.63 

.20 

.11 

.14 

.06 

.93 

20.  Geography 

.89 

.03 

.16 

.05 

-.00 

.01 

.22 

.86 

21.  Medicine  and  Dent. 

.13 

.10 

.29 

-.11 

.81 

.15 

-.00 

.80 

22.  Therapy  and  Counsel 

.76 

-.02 

.44 

.10 

.13 

.11 

-.27 

.88 

23.  Education  and  Train. 

.51 

.10 

.36 

.42 

.46 

.21 

-.04 

.82 

24.  English  Language 

.55 

.03 

.37 

.68 

.05 

.12 

.01 

.91 

25.  Foreign  Language 

.75 

.14 

.26 

-.03 

-.03 

.19 

-.39 

.83 

26.  Fine  Arts 

.86 

.11 

-.11 

.14 

.10 

.04 

-.34 

.92 

27.  History  and  Archeol. 

.84 

.16 

.15 

.13 

.27 

-.12 

.18 

.89 

28.  Philosophy  and  Theol. 

.66 

.01 

.60 

.10 

.20 

.25 

-.02 

.91 

29.  Public  Safety 

.22 

.22 

.86 

-.00 

.01 

-.13 

-.14 

.88 

30.  Law,  Govt.,  & Juris. 

.16 

.10 

.86 

.18 

.16 

.08 

;25 

.90 

31.  Telecomm. 

.35 

.50 

.34 

-.09 

-.25 

.07 

.46 

.77 

32.  Comm,  and  Media 

.59 

.11 

.36 

.46 

.17 

.23 

.35 

.90 

33.  Transportation 

.46 

.32 

.15 

-.05 

.01 

.13 

-.25 

.42 

Percent  of  Variance  35 

16 

11 

8 

6 

5 

3 

Eigenvalue  11.71 

5.32 

3.57 

2.70 

2.12 

1.65 

1.09 

Note.  N - 33.  The  correlation  matrix  was  based  on  means  calculated  at  the  occupation  level.  F ■ = 
Arts  & Humanities,  F2  = Science  & Technology,  F3  = Law  Enforcement,  F4  = Clerical,  F5  = 

Medicine,  F6  = Business  Administration,  F7  = High  Technology.  These  loadings  are  based  on  an 
orthogonal  varimax  rotation. 
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Descriptor  Means  and  Standard  Deviations  on  the  Level  Scale  on  Six  Example  Occupations:  Knowledges 
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Table  4- 12a  (continued) 

Descriptor  Means  and  Standard  Deviations  on  the  Level  Scale  on  Six  Example  Occupations:  Knowledges 
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Table  4- 12b 

Descriptor  Means  and  Standard  Deviations  on  the  Importance  Scale  on  Six  Example  Occupations:  Knowledges 
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Table  4- 12b  (continued) 

Descriptor  Means  and  Standard  Deviations  on  the  Importance  Scale  on  Six  Example  Occupations:  Knowledges 
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Table  4-13 

Rotated  Correlations  Between  Discriminating  Variables  and  Canonical  Discriminant  Functions  for  the  Level  Scale:  Knowledges 
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Table  4-13  (continued) 

Rotated  Correlations  Between  Discriminating  Variables  and  Canonical  Discriminant  Functions  for  the  Level  Scale:  Knowledges 
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Table  4- 14a 

Comparison  Between  Incumbent  and  Analyst  Descriptive  Statistics  Across  All  Occupations  and 
Reliability  Estimates  for  Rated  Differences  Between  Occupations  for  the  Level  Scale: 
Knowledges 


Incumbent Analyst 


Descriptor 

M 

SD 

Ik 

M 

SD 

Ik 

t 

F 

lia 

d2 

1 . Admin,  and  Mgmt. 

2.35 

2.16 

.88 

1.90 

1.85 

.96 

3.41* 

1.37 

.69 

1.43 

2.  Clerical  . 

3.30 

2.55 

.95 

2.80 

1.64 

.94 

3.71* 

2.43* 

.77 

.83 

3.  Economics  and  Acct. 

1.82 

2.21 

.84 

1.75 

1.65 

.96 

.56 

1.79 

.62 

1.20 

4.  Sales  and  Marketing 

1.41 

2.10 

.87 

1.38 

1.62 

.94 

.31 

1.67 

.77 

.82 

5.  Cust.  and  Pers.  Svc. 

2.75 

2.48 

.80 

2.86 

1.79 

.85 

-.84 

1.93 

.54 

1.44 

6.  Personnel  and  HR 

2.17 

2.25 

.82 

1.61 

1.70 

.95 

4.27* 

1.74 

.61 

1.34 

7.  Production  and  Proc. 

1.21 

2.08 

.70 

.96 

1.48 

.85 

2.18* 

1.97 

.70 

.86 

8.  Food  Production 

.68 

1.70 

.54 

.37 

.90 

.84 

3.71* 

3.60* 

.33 

.74 

9.  Computers  and  Elect. 

2.43 

2.06 

.83 

2.01 

1.51 

.85 

3.58* 

1.87 

.81 

.63 

10  Engineering  and  Tech. 

1.10 

2.00 

.76 

1.25 

1.39 

.92 

-1.37 

2.06* 

.84 

.43 

11.  Design 

1.14 

2.03 

.63 

.79 

1.39 

.84 

3.15* 

2.12* 

.62 

.87 

12.  Building  and  Constr. 

1.06 

2.09 

.81 

.61 

1.23 

.89 

4.17* 

2.86* 

.63 

1.28 

13.  Mechanical 

1.21 

2.04 

.85 

2.03 

1.73 

.95 

-6.51* 

1.38 

.84 

.96 

14.  Mathematics 

2.93 

1.82 

.82 

2.52 

1.26 

.94 

4.07* 

2.11* 

.56 

1.23 

15.  Physics 

.91 

1.82 

.75 

1.12 

1.27 

.92 

-2.09* 

2.04* 

.79 

.48 

16.  Chemistry 

1.21 

2.00 

.83 

1.43 

1.62 

.96 

-1.87 

1.53 

.75 

1.00 

17.  Biology 

.87 

1.92 

.81 

1.07 

1.54 

.96 

-1.72 

1.55 

.86 

-.47 

18.  Psychology 

2.49 

2.38 

.88 

2.14 

1.61 

.90 

2.70* 

2.18* 

.71 

1.08 

19.  Sociology  and  Anthro. 

1.22 

1.89 

.82 

1.30 

1.18 

.86 

-.84 

2.58* 

.78 

.41 

20.  Geography 

1.31 

1.98 

.62 

1.35 

1.28 

.88 

-.47 

2.37* 

.61 

.70 

21.  Medicine  and  Dent. 

1.17 

2.02 

.82  ' 

1.11 

1.35 

.95 

.56 

2.24* 

.79 

.66 

22.  Therapy  and  Couns. 

1.39 

2.13 

.85 

1.24 

1.57 

.88 

1.27 

1.83 

.66 

1.12 

23.  Education  and  Tm. 

2.30 

2.31 

.81 

2.17 

1.99 

.96 

.94 

1.35 

.72 

1.41 

24.  English  Language 

3.18 

2.01 

.85 

2.73 

1.19 

.95 

4.37* 

2.86* 

.59 

1.02 

25.  Foreign  Language 

.86 

1.71 

.37 

.64 

.83 

.11 

2.72* 

4.23* 

.20 

.64 

26.  Fine  Arts 

.58 

1.52 

.53 

.40 

.87 

.89 

2.35* 

3.07* 

.52 

.58 

27.  History  and  Archeol. 

.71 

1.61 

.67 

.72 

1.10 

.74 

-.08 

2.16* 

.76 

.36 

28.  Philosophy  and  The. 

.99 

1.75 

.75 

.70 

.99 

.68 

3.28* 

3.15* 

.63 

.58 

29.  Public  Safety  and  Sec. 

1.71 

2.11 

.87 

2.09 

1.37 

.92 

-3.41* 

2.39* 

.79 

.70 

30.  Law,  Govt.,  and  Jurisp. 

1.54 

2.05 

.84 

2.11 

1.39 

.87 

-5.11* 

2.17* 

.70 

.96 

31.  Telecommunications 

1.42 

1.73 

.47 

1.23 

.96 

.73 

2.21* 

3.24* 

.13 

.79 

32.  Comm,  and  Media 

1.72 

1.88 

.68 

1.88 

1.28 

.88 

-1.57 

2.14* 

.58 

.63 

33.  Transportation 

1.02 

1.67 

.64 

1.30 

1.35 

.90 

-2.84* 

1.54 

.60 

.72 

Note.  Incumbent  statistics  based  on  33  occupations  with  Knowledge  questionnaire  responses 
from  at  least  4 incumbents  (mean  number  of  incumbents  = 19.66,  median  =11,  harmonic  mean  = 
9.18).  Analyst  statistics  are  based  on  the  same  33  occupations  with  Knowledge  questionnaire 
responses  from  at  least  6 analysts  (mean  number  of  analysts  = 9.81,  median  = 12,  harmonic  mean 
= 8.30). 
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T able  4- 1 4a  (continued) 

Comparison  Between  Incumbent  and  Analyst  Descriptive  Statistics  Across  All  Occupations  and 
Reliability  Estimates  for  Rated  Differences  Between  Occupations  for  the  Level  Scale: 
Knowledges  ' 

The  estimate  of  reliability  was  obtained  by  calculating  the  intraclass  correlation  fork  ratings 

across  occupations:  ICC(l,k)  = fBMS-WMSl/BMS  (Shrout  & Fleiss,  1979),  where  k is  the 

harmonic  mean  of  the  number  of  ratings  provided  on  each  occupation. 

The  t statistic  tests  for  differences  in  the  incumbent  and  analyst  group  means. 

The  F statistic  tests  for  differences  in  the  incumbent  and  analyst  group  standard  deviations. 

The  r,a  correlation  indicates  the  degree  of  relationship  between  incumbent  and  analyst  mean 

occupations  ratings. 

,2  . 

The  d statistic  indicates  the  squared  differences  between  incumbent  and  analyst  mean 
occupations  ratings. 

< .05 


Table  4- 14b 


Reliability  Estimates  for  Rated  Differences  Between  Occupations  for  the  ImDortance  Scale- 

Knowledges 

Descriptor 

Incumbent 

Analyst 

M 

SD 

Ik 

M 

SD 

Ik 

t 

F 

Da 

d2 

1.  Admin,  and  Mgmt. 

2.51 

1.45 

.88 

2.28 

1.49 

. .96 

2.25* 

1.06 

.69 

.77 

2.  Clerical 

2.83 

1.45 

.93 

2.86 

1.32 

.94 

-.31 

1.22 

.72 

.67 

3.  Economics  and  Acct. 

2.09 

1.33 

.83 

2.14 

1.29 

.96 

-.56 

1.07 

.70 

.59 

4.  Sales  and  Marketing 

1.81 

1.22 

.88 

1.90 

1.26 

.95 

-1.06 

1.06 

.79 

.39 

5.  Cust.  and  Pers.  Svc. 

2.63 

1.47 

.76 

3.25 

1.47 

.86 

-6.17* 

1.00 

.42 

1.39 

6.  Personnel  and  HR 

2.27 

1.36 

.79 

1.81 

1.22 

.95 

5.26* 

1.25 

.65 

.68 

7 .Production  and  Proc. 

1.64 

1.13 

.71 

1.54 

1.02 

.89 

1.29 

1.22 

.66 

.43 

8.  Food  Production 

1.35 

.88 

.69 

1.25 

.79 

.83 

1.77 

1.25 

.40 

.27 

9 .Computers  and  Elect. 

2.50 

1.36 

.85 

2.17 

1.18 

.86 

3.95* 

1.33 

.76 

.35 

10.  Engineering  and  Tech. 

1.57 

1.07 

.78 

1.72 

1.10 

.93 

-2.05* 

1.06 

.88 

.18 

11.  Design 

1.56 

1.06 

.60 

1.30 

.72 

.90 

4.64* 

2.19* 

.58 

.35 

12.  Building  and  Constr. 

1.53 

1.08 

.84 

1.32 

.83 

.92 

3.34* 

1.67 

.66 

.38 

13.  Mechanical 

1.60 

1.06 

.87 

2.23 

1.33 

.96 

-7.46* 

1.56 

.74 

.84 

14.  Mathematics 

2.82 

1.14 

.79 

3.02 

1.07 

.88 

-2.63* 

1.13 

.42 

.63 

15.  Physics 

1.46 

.95 

.78 

1.62 

.99 

.90 

-2.43* 

1.08 

.68 

.33 

16.  Chemistry 

1.66 

1.13 

.83 

1.85 

1.20 

.95 

-2.46* 

1.11 

.72 

.52 

17.  Biology 

1.44 

1.02 

.83 

1.62 

1.19 

.96 

-2.31* 

1.37 

.78 

.41 

18.  Psychology 

2.35 

1.36 

.86 

2.40 

1.22 

.88 

-.52 

1.24 

.69 

.40 

19.  Sociology  and  Anthro. 

1.60 

1.01 

.83 

1.70 

.95 

.86 

-1.52 

1.13 

.85 

.11 

20.  Geography 

1.65 

1.04 

.64 

1.81 

1.10 

.92 

-2.25* 

1.12 

.61 

.42 

21.  Medicine  and  Dent. 

1.73 

1.27 

.86 

1.72 

1.28 

.96 

.07 

1.01 

.78 

.56 

22.  Therapy  and  Couns. 

1.77 

1.20 

.85 

1.72 

1.15 

.90 

.57 

1.09 

.59 

.56 

23.  Education  and  Tm. 

2.23 

1.28 

.81 

2.42 

1.44 

.95 

-2.05* 

1.27 

.73 

.69 

24.  English  Language 

2.98 

1.23 

.81 

3.31 

1.06 

.91 

-4.27* 

1.34 

.74 

.49 

25.  Foreign  Language 

1.41 

.84 

.43 

1.19 

.51 

.53 

4.97* 

2.74* 

.33 

.20 

26.  Fine  Arts 

1.28 

.75 

.64 

1.24 

.65 

.89 

.89 

1.34 

.59 

.20 

27.  History  and  Archeol. 

1.35 

.86 

.72 

1.22 

.62 

.90 

2.73* 

1.89 

.67 

.25 

28.  Philosophy  and  The. 

1.49 

.93 

.77 

1.30 

.67 

.60 

3.67* 

1.90 

.38 

.27 

29.  Public  Safety  and  Sec. 

2.10 

1.35 

.84 

2.55 

1.23 

.89 

-5.15* 

1.21 

.76 

.45 

30.  Law,  Govt.,  and  Jurisp. 

1.90 

1.24 

.84 

2.30 

1.20 

.91 

-4.80* 

1.07 

.71 

.49 

31.  Telecommunications 

2.08 

1.15 

.54 

1.95 

1.01 

.87 

1.75 

1.30 

.44 

.39 

32.  Comm,  and  Media 

2.00 

1.13 

.61 

2.32 

1.13 

.79 

-4.17* 

1.01 

.59 

.36 

33.  Transportation 

1.57 

.97 

.65 

1.85 

1.20 

.92 

-3.61* 

1.53 

.59 

.57 

Note.  Incumbent  statistics  based  on  33  occupations  with  Knowledge  questionnaire  responses 
from  at  least  4 incumbents  (mean  number  of  incumbents  = 19.66,  median  =11,  harmonic  mean  = 
9. 18).  Analyst  statistics  are  based  on  the  same  33  occupations  with  Knowledge  questionnaire 


Table  4- 14b  (continued) 

gompareon  Setween  Incumbent  and  Analyst  Descriptive  Statistics  Across  All  Occupations  anH 
pliability  Estimates  for  Rated  Differences  Between  Occupations  for  the  Importance  Scale: 


reJ>°“SeS  fr0m  at  least  6 (mean  number  of  analysts  = 9.81,  median  = 12,  harmonic  mean 

The  estimate  of  reliability  was  obtained  by  calculating  the  intraclass  correlation  for  k ratines 
across  occupations:  ICC(l,k)  = mS-WMS]mS  (Shrout  & Fleiss,  1979),  where  k is  the 
harmonic  mean  of  the  number  of  ratings  provided  on  each  occupation. 

The  t statistic  tests  for  differences  in  the  incumbent  and  analyst  group  means. 

The  F statistic  tests  for  differences  in  the  incumbent  and  analyst  group  standard  deviations. 

The  rja  correlation  indicates  the  degree  of  relationship  between  incumbent  and  analyst  mean 
occupations  ratings. 

The  d statistic  indicates  the  squared  differences  between  incumbent  and  analyst  mean 
occupations  ratings. 

*E  < .05 


Table  4-15 

Comparison  of  Incumbent  and  Analyst  Descriptive  Statistics  Across  All  Occupations  for  the 
Level  Scale  Rescored  Dichotomouslv:  Knowledges 


Incumbent Analyst 


Descriptor 

M 

SD 

Ik 

M 

SD 

Ik 

t 

1.  Admin,  and  Mgmt. 

.62 

.49 

.83 

.69 

.46 

.74 

-2.30* 

2.  Clerical 

.72 

.45 

.89 

.93 

.26 

.54 

-7.64* 

3.  Economics  and  Acct. 

.49 

.50 

.83 

.67 

.47 

.87 

-5.39* 

4.  Sales  and  Marketing 

.37 

.48 

.83 

.56 

.50 

.77 

-5.50* 

5.  Cust.  and  Pers.  Svc. 

.64 

.48 

.67 

.85 

.36 

.78 

-7.05* 

6.  Personnel  and  HR 

.57 

.50 

.78 

.60 

.49 

.68 

-1.04 

7.  Production  and  Proc. 

.30 

.46 

.64 

.40 

.49 

.68 

-3.30* 

8.  Food  Production 

.16 

.37 

.71 

.20 

.40 

.84 

-1.50 

9.  Computers  and  Elect. 

.66 

.47 

.81 

.77 

.42 

.50 

-3.47* 

10.  Engineering  and  Tech. 

.29 

.45 

.76 

.60 

.49 

.74 

-9.75* 

11.  Design 

.29 

.45 

.66 

.33 

.47 

.38 

-1.40 

12.  Building  and  Constr. 

.24 

.42 

.82 

.30 

.46 

.82 

-2.25* 

13.  Mechanical 

.31 

.46 

.80 

.80 

.40 

.53 

-16.11* 

14.  Mathematics 

.84 

.37 

.79 

.97 

.16 

.32 

-6.34* 

15.  Physics 

.26 

.44 

.80 

.60 

.49 

.80 

-11.05* 

16.  Chemistry 

.33 

.47 

.79 

.61 

.49 

.82 

-8.58* 

17.  Biology 

.21 

.41 

.82 

.47 

.50 

.82 

-8.57* 

18.  Psychology 

.60 

.49 

.69 

.78 

.41 

.76 

-5.67* 

19.  Sociology  and  Anthro. 

.35 

.48 

.80 

.65 

.48 

.64 

-9.29* 

20.  Geography 

.38 

.49 

.69 

.64 

.48 

.75 

-7.94* 

21.  Medicine  and  Dent. 

.34 

.47 

.80 

.54 

.50 

.76 

-6.14* 

22.  Therapy  and  Couns. 

.37 

.48 

.84 

.49 

.50 

.83 

-3.64* 

23.  Education  and  Tm. 

.59 

.49 

.72 

.72 

.45 

.74 

-4.20* 

24.  English  Language 

.83 

.38 

.80 

.98 

.12 

.15 

-7.24* 

25.  Foreign  Language 

.27 

.44 

.57 

.46 

.50 

.77 

-6.00* 

26.  Fine  Arts 

.16 

.37 

.63 

.21 

.41 

.88 

-1.79 

27.  History  and  Archeol. 

.20 

.40 

.67 

.35 

.48 

.51 

-5.08* 

28.  Philosophy  and  The. 

.28 

.45 

.76 

.45 

.50 

.34 

-5.57* 

29.  Public  Safety  and  Sec. 

.51 

.50 

.70 

.86 

.34 

.47 

-11.47* 

30.  Law,  Govt.,  and  Jurisp. 

.43 

.50 

.82 

.83 

.38 

.62 

-12.70* 

31.  Telecommunications 

.62 

.49 

.60 

.73 

.44 

.52 

-3.63* 

32.  Comm,  and  Media 

.55 

.50 

.68 

.80 

.40 

.68 

-7.63* 

33.  Transportation 

.35 

.48 

.61 

.62 

.49 

.72 

-8.49* 

HqISi  Incumbent  statistics  are  based  on  35  occupations  with  Skills  questionnaire  responses  from 
at  least  4 incumbents  (mean  number  of  incumbents  = 18.51,  median  =12,  harmonic  mean  = 
9.01).  Analyst  statistics  are  based  on  the  same  35  occupations  with  Skills  questionnaire  responses 
from  at  least  6 analysts  (mean  number  of  analysts  = 10.29,  median  = 12.0,  harmonic  mean  = 
8.66). 


O 
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Table  4- 1 5 (continued) 

Comparison  of  Incumbent  and  Analyst  Descriptive  Statistics  Across  All  Occupations  for  the 
Level  Scale  Rescored  Dichotomouslv:  Knowledges 


The  estimate  of  reliability  was  obtained  by  calculating  the  intraclass  correlation  for  k ratings 
across  occupations:  ICC(l,k)  = HBMS-WMS1/BMS  (Shrout  & Fleiss,  1979),  where  k is  the 
harmonic  mean  of  the  number  of  ratings  provided  on  each  occupation. 

The  t statistic  tests  for  differences  in  the  incumbent  and  analyst  group  means 
*2  < .05 


Table  4-16 

Principal  Components  Analysis  Pattern  Matrix  for  the  Analyst  Level  Scale:  Knowledges 


Factor 


Descriptor 

FI 

F2 

F3 

F4 

F5 

F6 

F7 

Communality 

1.  Admin,  and  Mgmt. 

.88 

.10 

.15 

.27 

.14 

.00 

-.05 

.89 

2.  Clerical 

.37 

-.38 

-.08 

-.14 

-.08 

.57 

.44 

.84 

3.  Economics  and  Acct. 

.84 

-.04 

-.19 

-.19 

.06 

.12 

.19 

.83 

4.  Sales  and  Marketing 

.67 

-.08 

-.36 

.25 

.01 

-.27 

.02 

.71 

5.  Cust.  and  Pers.  Svc. 

.43 

-.58 

.27 

.42 

.23 

-.06 

-.15 

.85 

6.  Personnel  and  HR 

.88 

-.09 

.13 

-.04 

.06 

-.07 

-.20 

.85 

7.  Production  Proc. 

.30 

.66 

-.24 

-.19 

.02 

-.46 

.13 

.83 

8.  Food  Production 

.13 

-.13 

-.03 

,14 

-.17 

-.81 

.14 

.75 

9.  Computers  and  Elect. 

.40 

.13 

-.06 

.07 

-.32 

.72 

.20 

.85 

10.  Engineering  and  Tech. 

.00 

.91 

.13 

-.21 

.08 

.05 

-.18 

.94 

11.  Design 

.17 

.84 

-.16 

.12 

-.10 

.16 

-.02 

.81 

12.  Building  and  Constr. 

.01 

.56 

-.24 

-03 

-.07 

-.04 

-.57 

.70 

13.  Mechanical 

-.42 

.65 

.01 

-.33 

.03 

-.03 

-.43 

.88 

14.  Mathematics 

.71 

.45 

-.07 

.18 

-.16 

.31 

.29 

.94 

15.  Physics 

-.22 

.92 

.20 

.01 

.13 

-.06 

.00 

.95 

16.  Chemistry 

-.05 

.66 

.66 

.00 

-.08 

-.09 

.09 

.89 

17.  Biology 

.04 

.22 

.86 

.18 

-.15 

-.08 

.20 

.89 

18.  Psychology 

.60 

-.21 

.53 

.39 

.28 

.04 

-.08 

.92 

19.  Sociology  and  Anth. 

.44 

-45 

.34 

.64 

.44 

.10 

-.09 

.94 

20.  Geography 

.06 

.07 

.00 

.43 

.82 

.10 

.25 

.94 

21.  Medicine  and  Dent. 

-.03 

-.14 

.95 

.04 

.04 

.08 

-.06 

.93 

22.  Therapy  and  Counsel. 

.27 

-.14 

.80 

.39 

.14 

.11 

-.07 

.91 

23.  Education  and  Train. 

.70 

.00 

.28 

.55 

-.02 

.05 

-.11 

.88 

24.  English  Language 

.72 

.00 

.30 

.35 

-.07 

.38 

.22 

.92 

25.  Foreign  Language 

.39 

-.21 

.40 

•49 

.46 

-.17 

.11 

.84 

26.  Fine  Arts 

.04 

-.18 

-.07 

.80 

-.12 

-.26 

-.15 

.79 

27.  History  and  Archeol. 

.08 

.07 

.16 

.85 

.15 

.14 

.19 

.83 

28.  Philosophy  and  Theol. 

.35 

-.06 

.30 

.75 

.33 

.02 

.02 

.88 

29.  Public  Safety 

.27 

.22 

.45 

-.02 

.60 

.03 

-.36 

.81 

30.  Law,  Govt.,  and  Juris. 

.66 

.00 

.38 

.06 

.57 

.03 

.08 

.90 

31.  Telecommunications 

.24 

-.22 

.07 

.12 

.45 

.65 

.06 

.75 

32.  Comm,  and  Mdia 

.80 

-.16 

.07 

.29 

.04 

.32 

.10 

.87 

33.  Transportation 

-.08 

-.02 

-.20 

.01 

.82 

.04 

-.10 

.74 

Percent  of  Variance 

33 

15 

13 

9 

8 

6 

3 

Eigenvalue 

10.83 

5.10 

4.20 

2.79 

2.53 

1.89 

1.07 

The  correlation  matrix  was  based  on  means  calculated  at  the  occupation  level.  FI  = General 
Management,  F2  = Engineering,  F3  = Allied  Health  Services,  F4  = Art  and  Humanities,  F5  = Law 
Enforcement/Public  Safety,  F6  = High  Technology,  F7  = Clerical.  These  loadings  are  based  on  an 
orthogonal  varimax  rotation. 
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Chapter  5 

Education,  Training,  Experience,  and  Licensure/Certification: 
Evidence  for  the  Reliability  and  Validity  of  Measures 

Lance  E.  Anderson 
American  Institutes  for  Research 


Education,  training,  experience,  and  licensure/certification  are  all  types  of  information 
that  describe  the  preparation  needed  before  entering  an  occupation.  They  are  ways  of  acquiring, 
or  demonstrating  the  acquisition  of  knowledges  and  skills.  As  such,  this  information  has 
traditionally  been  of  great  interest  to  career  and  vocational  counselors,  vocational  rehabilitation 
counselors,  job  seekers,  and  employers. 

Education,  training,  and  experience  are  commonly  recognized  as  mechanisms  for 
acquiring  general  knowledge  and  basic  skills  (Halpem,  1994).  For  this  reason,  they  have  and  will 
continue  to  have  a significant  role  in  the  selection  of  employees  (Dye  & Reck,  1988;  Monahan  & 
Muchinsky,  1983).  Workers  need  to  be  increasingly  better  prepared  to  deal  with  the  growing 
complexities  of  the  workplace  brought  on  by  tumultuous  changes  in  technology  and  realization 
of  a global  economy  (Goldstein  & Gilliam,  1990).  Therefore,  education,  training,  and  experience, 
as  ways  of  acquiring  knowledge  and  skills,  will  likely  become  more  important  in  the  future.  In 
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recognition  of  this,  various  programs  have  been  initiated  at  a federal  level  to  increase  the  link 
between  education  and  the  knowledges  and  skills  needed  to  succeed  in  the  workplace  (e.g.. 
Secretary's  Commission  on  Achieving  Necessary  Skills  (SCANS),  DOL,  1991).  Employers  are 
also  spending  more  money  on  training,  likely  due  to  the  growing  complexity  of  the  workplace 
and  the  waning  number  of  new  entrants  into  the  workforce  (Cascio  & Zammuto,  1987). 

As  education,  training,  and  experience  become  recognized  as  more  important, 
licensure/certification  programs  will  become  more  prevalent.  Scholars  and  blue-ribbon 
commissions  appointed  by  the  government  and  Congress  (e.g.,  Dertouzos,  Lester,  & Solow, 
1989,  U.S.  Congress,  Office  of  Technology  Assessment,  1990)  agree  that  licensure/certification 
of  employees  is  important  to  the  high  performance  workplace. 

The  0*NET  occupational  information  system  provides  information  on  occupations  for 
descriptors  pertaining  to  education,  training,  experience,  and  licensure/certification.  In  this 
chapter,  we  review  the  available  evidence  for  the  usefulness  of  these  descriptors,  beginning  with 
the  reliability  and  validity  of  the  measures  used  in  the  initial  0*NET  data  collection  effort.  We 
also  examine  the  implications  of  these  findings  for  refinement  and  extension  of  the  0*NET 
content  model.  Before  turning  to  the  relevant  data  however,  we  first  review  the  various 
taxonomies  and  the  procedures  used  to  collect  information  about  education,  training,  experience, 
and  licensure/certification. 

Background 

Taxonomies 

0*NET  measures  within  each  of  these  domains  are  based  on  a set  of  taxonomies 
proposed  by  Anderson  (1995).  These  taxonomies  were  based  in  general  on  an  examination  of  the 
needs  of  various  potential  users  of  the  0*NET,  an  examination  of  available  taxonomies  that  exist 
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relevant  to  these  domains,  and  a review  of  methods  commonly  used  by  researchers  to  collect  this 
type  of  information. 

Education 

Our  review  of  DOT  user  surveys  (Westat,  1993;  Campion,  M.,  Gowing,  M.,  Lancaster, 

A.,  & Pearlman,  K. , 1994)  indicated  that  the  following  information  is  important  to  users 
regarding  education  and  training: 

• the  amount  of  education  needed  to  enter  the  occupation,  including  the  degrees  and 

certificates  required;  and 

• the  type  of  education  needed  to  enter  the  occupation,  including 

• course  major,  or  instructional  program,  and 

• coursework,  or  courses  in  certain  subject  areas. 

Next,  we  explored  the  literature  for  taxonomies  relevant  to  the  user  needs.  Although  there 
are  various  education-oriented  taxonomies  described  in  the  literature  (e.g..  Bloom,  1956;  Gagne, 
1985),  we  found  one  set  of  taxonomies  relevant  to  user  needs.  This  set  of  taxonomies  is  called 
the  Classification  of  Instructional  Programs  (CIP;  U.S.  Department  of  Education,  1990). 

The  CIP  provides  classifications  of  education  by  type  and  amount.  The  "amount" 
taxonomy  found  in  the  CIP  classifies  education  into  categories  such  as  high  school, 
undergraduate,  and  graduate.  The  "type"  taxonomy  found  in  the  CIP  classifies  education 
according  to  major  area  of  study  (e.g..  Library  Science  [Descriptor  #2u].  Construction  Trades 
[Descriptor  #2k].  Engineering  [Descriptor  #2m]).  This  taxonomy  was  designed  to  include  all 
academic  and  occupation-specific  instructional  programs.  These  taxonomies  not  only  address 
both  kinds  of  user  needs,  they  are  also  crosswalked  to  the  current  DOT  and  the  Occupational 
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Employment  Statistics  job  families.  Therefore,  we  created  0*NET  measures  for  both 
taxonomies. 

We  also  needed  a taxonomy  to  describe  coursework/subjects  required  for  occupations. 
We  developed  this  taxonomy  based  on  a survey  of  courses  taught  by  teachers  (National  Center 
for  Education  Statistics,  1993)  and  an  examination  of  college  coursework  catalogs. 
Training/Experience 

This  taxonomy  includes  the  various  types  of  occupation-related  training  and  experiences. 
These  types  included  Related  Work  .Experience  (Descriptor  #7a),  On-Site  or  In-Plant  Training 
(Descriptor  #7b),  On-the  Job  Training  (Descriptor  #7c),  and  apprenticeships  (Descriptor  #7d). 
Related  work  experience  (Descriptor  #7a)  refers  to  experience  in  related  jobs.  Acceptable 
performance  in  some  jobs  requires  a certain  amount  of  experience  in  related  jobs.  For  example, 
many  managerial  occupations  require  a particular  number  of  years  of  experience  in  a related 
technical  job.  On-Site  or  In-Plant  Training  (Descriptor  #7b),  On-the  Job  Training  (Descriptor 
#7c),  and  apprenticeships  (Descriptor  #7d)  refer  to  training  experiences  that  occur  in  the  work 
context.  On-Site  or  In-Plant  Training  (Descriptor  #7b)  is  organized  classroom  study  required  and 
provided  by  an  employer.  On-the  Job  Training  (Descriptor  #7c)  is  when  an  individual  serves  as  a 
learner  or  trainee  on  the  job  under  the  instruction  of  a more  experienced  worker  (DOL,  1991). 
Finally,  apprenticeships  (Descriptor  #7d)  are  training  experiences  that  require  one  or  more  years 
of  On-the-Job  Training  (Descriptor  #7c)  through  work  experience  supplemented  by  related 
instruction.  Such  experience  is  often  required  before  one  can  be  considered  a qualified  and 
skilled  worker  (DOL,  1991). 
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Licensure/Certification 

The  following  ways  of  classifying  licensure/certification  programs  were  identified  as 
potentially  most  useful  to  0*NET  users: 

• name  of  the  license/certificate  relevant  to  the  occupation, 

• criteria  that  must  be  satisfied  to  obtain  the  license/certificate,  and 

• entities  that  require  the  license. 

The  first  two  types  of  information  focus  on  identifying  the  name  of  the  license/certificate 
relevant  to  the  occupation,  and  then  determining  the  basic  requirements  needed  to  achieve  and 
retain  licensure/certification.  This  data  would  be  of  obvious  interest  to  career/vocational 
counselors.  The  last  type  of  information  acknowledges  that  a license/certificate  may  be  required 
by  law,  the  employer,  or  a union  or  professional  organization. 

Sample  and  Measures 

A detailed  description  of  the  procedures  used  to  measure  education,  training,  experience 
and  licensure/certification  requirements  within  the  0*NET  system  has  been  provided  by 
Anderson  (1995).  We  used  a variety  of  measurement  scales  and  item  types  to  tap  the  various 
constructs.  Our  goal  was  to  collect  reliable  information  on  these  constructs  in  a manner  that 
would  place  the  least  burden  on  the  respondent.  Thus,  we  attempted  to  write  items  in  simple, 
exact  language. 

The  item  types  and  measurement  scales  are  listed  by  descriptor  in  Figure  5-1. 

The  sample  consisted  of  incumbents,  or  people  working  in  the  occupation,  each  of  whom 
was  asked  to  rate  the  requirements  of  his  or  her  job.  The  incumbent  sample  was  obtained  through 
a stratified  random  sampling  of  80  occupations  within  a stratified  random  sampling  of 
establishments  (i.e.,  employers).  A total  of  722  incumbents  returned  questionnaires  that  were  at 
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least  partially  completed.  From  these  data,  we  eliminated  occupations  with  fewer  than  four 
respondents.  Following  this  process,  598  cases  remained.  Figure  5-2  lists  the  34  occupations  and 
the  number  of  incumbents  who  completed  the  ratings.  Note  that  incumbents  whose  responses 
were  retained  are  employed  at  over  100  establishments.  We  used  these  data  for  most  of  our 
analyses. 

Results 

We  conducted  various  analyses  on  the  data  to  assist  in  evaluating  the  effectiveness  of 
these  descriptors.  Our  analyses  included  calculation  of  descriptive  statistics  for  each  descriptor, 
reliability  analyses  for  each  descriptor,  application  of  Analysis  of  Variance  (ANOVA)  procedures 
meant  to  examine  sources  of  variation  in  the  ratings,  reliability  analyses  for  each  major  area  of 
occupational  preparation,  intercorrelations  of  the  descriptors,  factor  analyses,  and  discriminant 
function  analyses. 

Before  presenting  the  results,  a note  on  the  descriptor  Instructional  Program  (Descriptor 
#2)  is  necessary.  The  Instructional  Program  (Descriptor  #2)  was  intended  as  a single  descriptor 
(Anderson,  1995).  The  purpose  of  the  descriptor  was  to  indicate  the  instructional  program 
required  for  an  occupation.  It  was  not  intended  to  gather  information  on  the  relevance  of  42 
separate  instructional  programs.  However,  to  facilitate  presentation  of  the  descriptive  statistics  on 
the  descriptor,  and  to  allow  for  an  evaluation  of  the  descriptor  as  42  separate  descriptors,  we 
present  statistics  42  separate  descriptors  in  many  of  the  tables  that  follow.  For  purposes  of 
presenting  these  results,  each  of  the  42  programs  is  defined  as  a dichotomous  variable  with  0 
indicating  "not  checked",  and  1 indicating  "checked". 

Descriptive  Statistics 
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Table  5-1  presents  the  basic  descriptive  statistics  obtained  for  each  scale.  This  table 
presents  the  overall,  cross-occupation  mean  and  standard  deviation  of  the  ratings  where 
occupations  were  treated  as  the  unit  of  analysis.  The  associated  interrater  agreement  coefficients 
and  standard  errors  of  measurement  are  also  presented. 

The  descriptive  statistics  are  consistent  with  the  results  we  expected  given  the 
occupations  included  in  our  sample.  The  results  on  general  level  of  education  are  typical~the 
mean  indicates  that  the  average  amount  of  education  is  somewhere  between  "High  School 
Diploma"  and  "Some  College  Courses."  In  fact,  over  50  percent  of  our  respondents  indicated  that 
the  general  level  of  education  required  for  their  occupation  is  High  School  or  less.  This  is  in  line 
with  the  general  level  of  education  found  in  the  U.S.  (National  Center  for  Education  Statistics, 
1993). 

The  descriptive  statistics  on  Instructional  Program  Requirement  (Descriptor  #2)  indicate 
that  43%  of  the  sample  endorsed  No  Specific  Major  (Descriptor  #2pp).  Also,  26%  of  the  sample 
endorsed  more  than  one  instructional  program.  The  responses  also  appear  to  be  consistent  with 
the  occupations  in  the  sample.  Note  that  at  least  10  percent  of  the  sample  indicated  that  their 
occupations  required  a major  in  computer  sciences,  education,  or  health  professions.  This  is  not 
surprising  when  one  considers  that  the  sample  included  Computer  Programmers,  Education 
Administrators,  Elementary  School  Teachers,  Registered  Nurses,  and  Medical  and  Clinical 
Laboratory  Technologists. 

Descriptors  pertaining  to  Training/Experience  and  Licensure/Certification  also  provided 
results  in  line  with  our  expectations.  In  terms  of  the  different  types  of  experience.  Related  Work 
Experience  (Descriptor  #7a)  was  viewed  as  required  more  often  than  the  other  types  of 
experience,  with  Apprenticeship  Experience  (Descriptor  #7d)  being  reported  as  the  least 
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frequently  required  type  of  experience.  Licensure/certification  was  reported  as  required  nearly 
thirty  percent  of  the  time.  The  most  frequently  reported  requirement  for  a license  was 
examination.  A license  was  most  frequently  required  by  the  employer,  the  law,  and  a 
union/association,  respectively. 

Reliability 

We  examined  the  reliability  of  each  descriptor  score  by  calculating  an  intraclass 
correlation  for  each  of  the  ratings  across  occupations  (Shrout  & Fleiss,  1979).  With  this  type  of 
intraclass  statistic,  reliability  increases  as  the  between  occupations  variance  is  greater  and  the 
within  occupations  variance  is  lower.  These  interrater  agreement  coefficients  are  based  on  a 
harmonic  mean  of  9.04  raters  per  occupation  and  are  presented  in  Table  5-1,  along  with  the 
associated  standard  errors  of  measurement. 

The  reliabilities  for  most  of  the  descriptors  were  good.  The  descriptor  General  Education 
Level  (Descriptor  #1)  had  a reliability  of  .97.  Most  of  the  descriptors  pertaining  to 
Training/Experience  and  Licensure/Certification  had  acceptable  reliabilities  (i.e.,  greater  than 
.75).  The  exceptions  to  this  include  Requirements  to  Obtain  a License:  Graduate  Degree 
(Descriptor  #5b),  which  has  a low  mean  and  low  variance  in  this  sample,  and  Experience:  On- 
Site  Training  Experience  (Descriptor  #7b),  a descriptor  that  likely  varies  with  different 
organizational  training  expectations,  and  thus  varies  within  occupation. 

When  analyzed  as  separate  descriptors.  Instructional  Program  (Descriptor  #2)  provided 
reliabilities  that  varied  greatly.  Some  of  these  descriptors  (e.g..  Health  Professions  [Descriptor 
#2q],  Mechanics/Repairers  [Descriptor  #2x],  and  Law/Legal  Studies  [Descriptor  #2s])  had 
reliabilities  above  .90,  while  others  had  reliabilities  below  .10  (e.g..  Social  Sciences/History 
[Descriptor  #2kk].  Science  Technologies  [Descriptor  #2jj],  and  Philosophy/Religion  [Descriptor 
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#2cc]).  However,  over  half  of  these  descriptors  had  reliabilities  less  than  .60.  It  is  of  note  that 
descriptors  with  particularly  low  reliabilities  also  had  low  mean  scores  and  low  standard 
deviations.  It  is  likely  that  with  some  of  these  descriptors  that  have  such  an  obvious  tie  to  specific 
occupations  (e.g..  Architecture  [Descriptor  #2c]),  reliabilities  would  increase  if  these  occupations 
were  part  of  the  sample. 

Overall,  these  initial  analyses  indicate  that  virtually  all  independent  descriptors  of 
occupational  preparation  yielded  adequate,  consistent  descriptions  of  occupational  requirements, 
while  Instructional  Program  (Descriptor  #2),  when  analyzed  as  separate  descriptors,  showed  great 
variance  in  the  size  of  the  reliability  coefficients.  We  obtained  these  results  with  relatively  small 
samples  of  incumbents  within  each  occupation.  As  the  size  of  interrater  agreement  coefficients 
depends  on  the  number  of  raters,  it  might  be  useful  to  examine  how  these  reliabilities  would  be 
affected  by  changes  in  the  number  of  raters.  Table  5-2  contains  the  results  of  analyses  where  we 
have  adjusted  the  reliability  coefficients  to  both  one  and  30  raters. 

The  pattern  of  the  results  for  different  numbers  of  raters  is  naturally  the  same,  while  the 
general  level  of  the  reliability  coefficients  changes.  Adjusting  the  reliability  coefficient  estimates 
for  30  raters  increases  the  reliability  coefficients  among  the  various  domains  so  that 
approximately  half  of  the  coefficients  eclipse  .90,  however,  there  are  still  some  coefficients 
below  .60.  Most  of  these  low  reliability  numbers  come  from  the  Instructional  Program 
(Descriptor  #2)  descriptors.  The  single  rater  reliability  coefficients  show  that  General  Education 
Level  (Descriptor  #1),  and  descriptors  pertaining  to  Licensure/Certification  continue  to  have 
acceptable  reliability  (most  of  them  are  above  .60),  while  the  reliabilities  for  Instructional 
Program  (Descriptor  #2)  are  generally  inadequate. 
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Analyses  of  Variance 

Another  way  one  might  examine  interrater  agreement,  in  addition  to  the  simple  one-way 
analyses  described  above,  is  by  examining  interrater  agreement  under  conditions  where 
descriptors  within  a domain  or  typology  are  treated  as  a repeated  measures  variable.  The  results 
obtained  in  this  analysis  are  presented  in  Tables  5-4a,  5-4b,  5-4c,  and  5-4d,  which  present  the 
results  of  these  analyses  for  the  following  areas:  Instructional  Program  (Descriptor  #2),  Subject 
Area  Education  Level  (Descriptor  #3),  Licensure/Certification,  and  Training/Experience. 

Across  all  five  analyses,  significant  differences  were  found  for  the  "occupations"  main 
effect,  the  "descriptor"  main  effect,  and  the  "descriptor  by  occupation"  interaction.  These  findings 
are  in  line  with  our  expectations.  They  indicate  that  the  mean  ratings  vary  across  occupations, 
across  descriptors,  and  that  the  pattern  of  ratings  across  descriptors  differs  from  occupation  to 
occupation. 

The  interrater  agreement  coefficients  resulting  from  these  analyses  are  presented  in  Table 
5-5,  along  with  the  single  rater  and  thirty  rater  agreement  coefficients  implied  by  these  analyses. 
These  overall  analyses  indicate  that  the  ratings  evidenced  good  interrater  agreement,  yielding 
coefficients  of  .78,  .74,  .85,  and  .79  for  the  Instructional  Program  (Descriptor  #2),  Level  of 
Education  Required  in  Specific  Subject  Areas  (Descriptor  #3),  Licensure/Certification,  and 
Training/Experience  descriptors  respectively.  Single  rater  agreement  coefficients  are  below  .40, 
and  with  30  raters  these  agreement  coefficients  would  lie  in  the  .90’s.  These  results  indicate  that 
approximately  10  ratings  per  occupation  would  provide  statistics  with  adequate  reliability,  while 
a single  rater  would  be  inadequate,  and  30  ratings  may  be  unnecessary. 

The  reliabilities  of  these  descriptors  when  examined  in  the  one-way  analyses  (Table  5-1) 
are  generally  lower  than  those  when  they  are  examined  in  the  repeated  measures  analysis  (Table 
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5-5).  The  reason  for  this  is  that  the  one-way  analyses  examine  differences  across  occupations 
within  descriptor  only,  while  the  repeated  measures  analyses  examine  differences  across 
occupations  across  all  descriptors  within  an  area.  Thus,  these  findings  are  not  in  conflict. 

The  applicability  of  these  different  reliability  estimates  depends  on  the  way  the  data  are 
being  used.  If  the  difference  of  individual  descriptor  scores  across  occupations  is  the  only 
difference  that  is  relevant,  then  the  reliability  estimates  associated  with  the  one-way  analyses 
apply.  If  the  mean  differences  between  the  patterns  of  descriptor  ratings  across  occupations  are- 
also  relevant,  then  the  repeated-measures  estimate  is  more  appropriate.  Thus  the  one-way 
estimates  apply  when  the  data  are  examined  on  a descriptor  by  descriptor  basis,  and  the  repeated 
measures  estimates  apply  when  examining  the  results  on  a set  of  descriptors.  For  example,  if  the 
query  were  "Is  the  instructional  program  of  Architecture  (Descriptor  #2c)  required  for  this  job?,” 
then  the  one-way  estimate  (.52)  would  be  relevant,  and  one  might  conclude  that  the  data  lacks 
sufficient  reliability  to  respond  to  the  query.  However,  if  the  query  were  "What  instructional 
program  is  relevant?,"  then  the  repeated  measures  estimate  (.78)  would  be  relevant,  and  one 
might  conclude  that  the  reliability  is  more  than  adequate  to  respond  to  the  query. 

Descriptor  Relationships 

Table  5-9  presents  the  correlations  among  the  descriptors  at  the  occupation  level, 
obtained  by  correlating  the  occupation  means  on  each  descriptor.  Table  5-10  presents  similar 
correlations  but  at  the  individual  level  using  four  individuals  per  occupation.  Because  the 
primary  concern  of  the  present  study  is  occupation  description,  we  will  focus  on  the  correlations 
obtained  at  the  occupation  level.  It  is  of  note  in  this  regard,  however,  that  the  pattern  of 
relationships  observed  at  the  individual  level  was  similar  to  that  obtained  at  the  occupation  level. 
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bearing  in  mind  the  higher  reliability  of  the  aggregate  ratings,  and  the  fact  the  lack  of  variance  on 
some  variables  at  the  individual  level  preclude  the  calculation  of  correlations. 

In  examining  the  occupational  correlations  among  descriptors,  it  is  clear  that  descriptors 
pertaining  to  a given  type  of  occupational  preparation  correlated  highly  with  one  another.  For 
example,  virtually  all  15  descriptors  pertaining  to  education  level,  with  the  exception  of  Subject 
Area  Education  Level:  Technical  Vocational  (Descriptor  #3a)  and  Subject  Area  Education  Level: 
Business  Vocational  (Descriptor  #3b),  had  intercorrelations  above  .50.  Most  descriptors 
pertaining  to  licensure/certification  also  had  high  positive  intercorrelations.  Finally,  all 
descriptors  pertaining  to  training/experience  had  positive  intercorrelations.  Training/Experience 
descriptors  that  include  a training  element  (e.g.,  On-Site  Training  Experience  [Descriptor  #7b], 
On-the-Job  Training  Experience  [Descriptor  #7c],  and  Apprenticeship  Experience  [Descriptor 
#7d])  had  intercorrelations  above  .50). 

Instructional  Program  (Descriptor  #2),  when  examined  as  separate  descriptors,  had 
largely  uninterpretable  intercorrelations.  The  low  correlations  are  likely  due  to  the  low  reliability 
associated  with  these  descriptors  when  viewed  individually,  which  in  turn  is  due  to  the 
understandably  very  low  endorsement  rates  of  the  descriptors,  owing  to  the  fact  that  we  had  just 
34  occupations.  Also  of  note  is  the  fact  that  more  than  50  percent  of  the  sample  indicated  that 
only  one  instructional  program  was  required.  As  any  correlation  among  these  descriptors  would 
be  driven  by  the  fraction  of  the  sample  where  more  than  one  instructional  program  is  required, 
these  correlations  may  not  reveal  the  true  relationship  among  the  various  instructional  programs. 
The  correlations  between  these  descriptors  and  other  descriptors  pertaining  to  Education, 
Training/Experience,  and  Licensure/Certification  were  also  uninterpretable. 

Factor  Structure 
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To  further  examine  the  pattern  of  relationships  among  the  various  descriptors,  we 
conducted  a principal  components  factor  analysis.  We  applied  this  factor  analysis  to  the 
occupation  level  correlations  among  all  of  the  descriptors  of  occupational  preparation,  save  for 
those  pertaining  to  Instructional  Program  (Descriptor  #2).  We  chose  not  to  include  Instructional 
Program  (Descriptor  #2)  as  separate  descriptors  because  of  the  generally  low  reliability  and  low 
base  rate  associated  with  these  descriptors.  Table  5-11  summarizes  the  results  obtained  in  this 
analysis  following  a varimax  rotation. 

Inspection  of  the  eigenvalues  and  a scree  test  indicated  that  a three  factor  solution 
provided  a clear  structure  for  summarizing  the  relationships  among  the  descriptors.  These  factors 
accounted  for  75%  of  the  total  variance  in  the  ratings. 

The  first  factor  extracted  in  this  analysis,  accounting  for  39%  of  this  variance,  was  labeled 
education.  General  Level  of  Education  (Descriptor  #1)  had  a high  loading  on  this  factor  (r  = .87). 
Also,  most  of  the  descriptors  pertaining  to  Subject  Area  Education  Level  (Descriptor  #3)  yielded 
sizable  loadings  on  this  factor.  For  example.  Oral  Communications  (Descriptor  #3 d)  (r  = ,93), 
Advanced  Math  (Descriptor  #3g)  (r  = .93),  English/Language  Arts  (Descriptor  #3c)  (r  = .92),  and 
Social  Sciences  (Descriptor  #31)  (r  = .92)  all  produced  sizable  loadings.  Two  descriptors 
pertaining  to  Subject  Area  Education  Level  (Descriptor  #3)  failed  to  load  highest  on  this  factor: 
Technical  Vocational  (Descriptor  #3a)  (r  = .34)  and  Business  Vocational  (Descriptor  #3b)  (r  = 
.32).  These  loadings  make  sense  however,  given  that  these  subject  areas  typically  result  in 
somewhat  lower  levels  of  education  (in  terms  of  years  completed)  and  tend  to  occur  for  different 
types  of  occupations. 

The  second  factor  extracted  in  this  analysis,  accounting  for  26%  of  the  total  variance,  was 
labeled  licensure.  Virtually  all  variables  pertaining  to  Licensure/Certification  had  high  loadings  (r 
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> .90)  on  this  factor.  Two  descriptors  pertaining  to  Licensure/Certification  did  not  have  a 
particularly  high  loading  on  this  factor:  Licenses:  Commercial  Vehicle  License  (Descriptor  #4a) 
(r  = .40),  and  Requirements  to  Obtain  a License:  Graduate  Degree  (Descriptor  #5b)  (r  = .41).  The 
first  of  these  two  descriptors  likely  had  a low  loading  due  to  the  different  nature  of  commercial 
vehicle  licenses  relative  to  other  licenses.  The  loading  for  Requirements  to  Obtain  a License: 
Graduate  Degree  (Descriptor  #5b)  was  likely  attenuated  by  the  low  base  rate  and  variance 
associated  with  this  type  of  degree.  Subject  Area  Education  Level:  Business  Vocational 
(Descriptor  #3b)  had  a moderately  high,  but  negative  loading  on  this  factor.  This  loading  likely 
occurred  because  that  subject  area  is  not  typically  associated  with  occupations  that  require 
licensure. 

The  third,  and  final,  factor  extracted  in  this  analysis  accounted  for  9%  of  the  variance  in 
descriptor  ratings.  This  factor,  labeled  training,  was  defined  in  terms  of  occupation  specific 
training,  including  Apprenticeship  Experience  (Descriptor  #7d)  (r  = .85),  On-the-Job  Training 
Experience  (Descriptor  #7c)  (r  = .82),  On-Site  Training  Experience  (Descriptor  #7b)  (r  = .79), 
and  Subject  Area  Education  Level:  Technical  Vocational  (Descriptor  #3a)  (r  = .68). 

Taken  as  a whole,  the  results  obtained  in  this  analysis  provide  some  initial  evidence  for 
the  meaningfulness  of  different  types  of  occupational  preparation.  Each  factor  describes  different 
types  of  occupational  preparation,  that  receive  different  emphasis  depending  on  the  occupation  at 
hand. 

While  the  factor  analysis  provides  information  as  to  the  relationships  among  the 
descriptors,  it  does  not  directly  address  the  issue  of  how  the  various  descriptors  differentiate 
occupations.  Thus,  it  should  not  be  assumed  that  these  factors  necessarily  provide  an  adequate 
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summary  system  when  our  concern  is  describing  the  similarities  and  differences  among 
occupations. 

Occupation  Differences 

Some  initial  evidence  bearing  on  the  ability  of  these  descriptors  to  capture  the  similarities 
and  differences  among  occupations  might  be  obtained  by  contrasting  the  mean  profile  of 
occupations  on  the  various  skill  scales.  Table  5-12  presents  the  means  and  standard  deviations  of 
the  descriptors  on  six  occupations  selected  to  reflect  distinct  types  of  employment:  (1)  General 
Managers  and  Top  Executives,  (2)  Computer  Programmers,  (3)  Registered  Nurses,  (4)  Police 
Patrol  Officers,  (5)  Janitors  and  Cleaners,  and  (6)  Maintenance  Repairers,  General  Utility. 

The  mean  scores  of  incumbents  on  these  descriptors  meet  with  our  expectations  of  the 
education,  training,  experience,  and  licensure/certification  requirements  for  these  occupations. 
According  to  the  mean  General  Level  of  Education  (Descriptor  #1),  these  six  occupations  rank  in 
terms  of  our  expectations,  with  General  Managers  and  Top  Executives  indicating  the  highest,  and 
Janitors  and  Cleaners  indicating  the  lowest  requirements.  Subject  Area  Education  Level 
(Descriptor  #3)  ratings  were  generally  higher  for  Computer  Programmers  and  Registered  Nurses 
than  they  were  for  the  other  occupations  included  in  the  six.  Ratings  on  individual  subject  areas 
also  met  with  our  expectations.  Computer  programmers  provided  the  highest  ratings  of  the  six  on 
Computer  Science  (Descriptor  #3i).  Advanced  Math  (Descriptor  #3g),  and  English/Language 
Arts  (Descriptor  #3c).  Registered  Nurses  received  the  highest  ratings  of  the  six  occupations  on 
Biological  Sciences  (Descriptor  #3jj.  Applied  Sciences  (Descriptor  #3k),  and  Physical  Sciences 
(Descriptor  #3h). 

Instructional  Program  Requirements  (Descriptor  #2)  were  generally  in  line  with  our 
expectations.  The  most  frequently  endorsed  Instructional  Program  Requirement  (Descriptor  #2) 
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for  General  Managers  and  Top  Executives  was  Management  and  Administration  (Descriptor  #2f) 
(53%),  for  Computer  Programmers,  it  was  Computer  Information  Sciences  (Descriptor  #2i) 
(83%),  for  Registered  Nurses,  it  was  Health  Professions  (Descriptor  #2q)  (97%),  for  Police  Patrol 
Officers,  it  was  Law  and  Legal  Studies  (Descriptor  #2s)  (82%),  for  Janitors  and  Cleaners  it  was 
No  Specific  Major  (Descriptor  #2pp)  (68%),  and  for  Maintenance  Repairers,  General  Utility,  it 
was  Mechanics/Repairers  (Descriptor  #2x)  (62%). 

Licensure/ certification  descriptors  were  generally  rated  highly  by  Registered  Nurses  and 
Police  Patrol  Officers,  with  100  percent  of  the  incumbents  in  both  occupations  indicating  that 
licensure/certification/registration  is  required  by  their  employer.  Licensure/certification  ratings 
were  moderately  high  for  Maintenance  Repairers,  General  Utility,  with  51  percent  of  incumbents 
indicating  that  licensure/certification/registration  is  required  by  their  employer.  These  ratings 
were  lowest  for  Computer  Programmers,  where  100%  of  incumbents  marked  "no"  in  response  to 
all  descriptors  pertaining  to  licensure  requirements. 

Finally,  in  terms  of  training  and  experience.  General  Managers  and  Top  Executives 
indicated  the  most  years  of  experience  required  in  related  occupations.  Maintenance  Repairers, 
General  Utility  provided  the  highest  ratings  on  three  descriptors  pertinent  to  training  (On-Site 
Training  Experience  [Descriptor  #7b],  On-the-Job  Training  Experience  [Descriptor  #7c],  and 
Apprenticeship  Experience  [Descriptor  #7d]). 

The  evidence  presented  above  does  lead  to  a noteworthy  conclusion.  Specifically,  it 
appears  that  the  various  descriptors  provide  a meaningful  description  of  the  similarities  and 
differences  among  occupations  in  terms  of  education,  training,  experience,  and 
licensure/certification.  This  point  is  of  some  importance  since  ultimately  a viable  descriptive 
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system  must  be  capable  of  capturing  and  accurately  reflecting  meaningful  differences  in  all  types 
of  occupational  preparation. 

Discriminant  Analyses 

While  the  foregoing  analysis  of  mean  differences  provided  information  on  how  the 
descriptors  differentiated  six  of  the  occupations,  we  also  wanted  to  determine  the  degree  to  which 
the  descriptors  differentiate  occupations  in  the  larger  sample  of  occupations.  We  conducted  a 
discriminant  analysis  to  address  this  issue.  This  analysis  included  all  of  the  descriptors  pertaining 
to  occupational  preparation,  save  for  Instructional  Program  (Descriptor  #2).  The  results  obtained 
in  this  analysis  are  presented  in  Table  5-13,  which  displays  both  the  loadings  on  the  discriminant 
functions  and  the  discriminating  variance  attributable  to  a given  descriptor. 

Discriminant  analysis  addresses  the  question,  “Do  the  Education,  Training,  Experience, 
and  Licensure  descriptors  differentiate  among  occupations?”  Table  5-13  presents  the  results  of  a 
discriminant  analysis  of  the  incumbent  respondents  from  the  34  occupations  included  in  the 
analyses  for  this  chapter.  Table  10-13  shows  (1)  the  correlation  between  each  descriptor  and  the 
first  six  rotated  discriminant  functions,  (2)  the  sum  of  the  squared  rotated  correlations  between 
the  discriminating  variables  (i.e.,  descriptors)  across  six  discriminant  functions  (EF2),  (3)  the 
proportion  of  variance  in  the  ratings  of  each  descriptor  accounted  for  by  occupations  (T|2),  (4)  the 
canonical  correlation  for  each  function  (Rc),  (5)  the  percent  of  variance  accounted  for  by  each 
function,  and  (6)  the  eigenvalue  for  each  function. 

Using  all  15  classification  functions,  not  just  the  six  shown  in  Table  5-13, 44%  of  the 
incumbents  were  correctly  reassigned  into  the  occupations  from  which  they  were  drawn.  This 
correct  assignment  rate  is  substantially  above  chance,  which  would  be  about  3%  for  34 
occupations,  if  all  the  occupations  were  of  equal  size. 
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The  best  discriminators  of  job  assignments  were  General  Education  Level  (Descriptor 
#1),  Licenses  (Descriptor  #4):  Commercial  Vehicle  License  (Descriptor  #4a),  and  Related  Work 
Experience  (Descriptor  #7a).  The  descriptors  pertaining  to  licensure/certification  were  generally 
moderately  useful  in  discriminating  between  job  assignments.  The  descriptors  pertaining  to 
Instructional  Program  (Descriptor  #2)  and  Subject  Area  Education  Level  (Descriptor  #3) 
demonstrated  less  usefulness  individually  for  differentiating  job  assignments  in  this  sample. 
However,  collectively,  the  educational  subject  area  description  did  show  utility  in  differentiating 
occupations  (the  fifth  function). 

The  findings  obtained  in  this  analysis  provide  some  clues  about  how  these  descriptors 
discriminate  occupations.  Inspection  of  the  descriptors  loading  on  the  six  functions  retained  in 
this  analysis  indicated  that  the  functions  appeared  to  reflect,  respectively,  (1)  licenses.  (2) 
education  level,  (3)  degree  required  to  obtain  licensure.  (4)  commercial  vehicle  license.  (5) 
coursework  required,  and  (6)  work  experience. 

Four  out  of  the  six  functions  were  dominated  by  a single  descriptor.  General  Education 
Level  (Descriptor  #1),  Post  Secondary  Degree  (Descriptor  #5a)  (a  requirement  to  obtain  a 
license).  Commercial  Vehicle  License  (Descriptor  #4a),  and  Related  Work  Experience 
(Descriptor  #7a)  each  loaded  singularly  with  one  of  the  four  functions.  The  first  function  had 
high  correlations  with  several  variables  related  to  licensure,  and  the  fifth  function  had  high 
correlations  with  several  variables  related  to  math  and  science  coursework. 

Conclusions 

In  this  section,  we  consider  the  findings  for  each  type  of  descriptor,  make  conclusions  as 
to  the  usefulness  of  these  descriptors,  and  recommend  improvements.  Before  we  turn  to  the 
conclusions  of  the  present  study,  certain  limitations  of  the  present  study  should  be  noted. 
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Some  of  these  limitations  are  relevant  to  every  domain  discussed  in  this  report,  and  thus 
have  been  discussed  in  detail  elsewhere.  We  summarize  them  here  in  the  form  of  two  caveats. 
First,  although  occupations  and  incumbents  were  sampled  in  a scientifically  sound  and 
representative  manner,  the  number  of  occupations  and  the  number  of  incumbents  within  those 
occupations  may  not  be  sufficient  to  make  firm  conclusions  about  these  descriptors.  Second, 
these  analyses  are  necessarily  limited  in  scope,  and  our  conclusions  may  change  as  additional 
analyses  are  done,  or  as  the  data  on  education,  training,  experience,  and  licensure/certification  is 
compared  with  the  data  from  other  domains. 

The  first  caveat  is  especially  relevant  with  education,  training,  experience,  and 
licensure/certification  descriptors,  because  some  of  these  descriptors  are  likely  to  be  particularly 
relevant  for  only  certain  occupations  (e.g..  Subject  Area  Education  Level:  Languages  [Descriptor 
#3e]).  If  these  occupations  are  not  in  the  sample,  then  the  variance  and  the  reliability  of  the  given 
descriptor  will  be  difficult  to  estimate  accurately. 

Even  bearing  these  caveats  in  mind,  we  believe  that  the  findings  obtained  in  the  present 
study  have  some  important  implications  for  the  assessment  of  education,  training,  experience, 
and  licensure/ certification  requirements.  To  begin,  it  appears  that  it  is  possible  to  formulate 
questions  and  rating  scales  that  can  be  used  to  obtain  reliable,  meaningful  data  on  these 
descriptors.  The  apparent  feasibility  of  assessing  these  descriptors  through  an  incumbent  survey 
strategy  is  noteworthy  because  it  represents  a relatively  low-cost  procedure  for  establishing 
education,  training,  experience,  and  licensure/certification  requirements. 

General  Education  Level 
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This  descriptor  was  meant  to  determine  the  education  level  required  for  the  occupation  at 
hand.  The  scale  for  this  descriptor  was  derived  from  the  CIP,  which  references  the  different 
education  levels.  Our  findings  on  this  descriptor  include: 

• descriptive  statistics  in  line  with  expectations, 

• high  reliability , 

• logical  correlations  with  other  education,  training,  experience,  and  licensure/certification 
descriptors, 

• prominence  in  defining  the  factor  of  Education  (factor  analytic  results),  and 

• usefulness  in  differentiating  occupations. 

These  findings  provide  generally  positive  evidence  for  retaining  the  item  as  written.  Note, 
however,  that  this  conclusion  is  based  on  the  premise  that  the  level  of  education  required  is  the 
relevant  level  of  analysis.  If  it  is  determined  that  users  may  be  more  interested  in  the  relevance  of 
each  of  the  levels  of  education,  then  a change  in  the  item  might  be  warranted.  This  change  would 
amount  to  splitting  the  item  into  eleven  different  descriptors,  each  of  which  evaluates  the 
importance  of  obtaining  education  at  the  given  level.  This  will  naturally  provide  more 
information,  and  thus  increase  overall  reliability  and  validity,  but  it  would  come  at  the  cost  of 
adding  items  to  an  already  lengthy  survey. 

Educational  Subject  Area 

These  descriptors  were  meant  to  determine  the  amount  of  formal  education  required  in 
each  of  15  different  subject  areas.  The  scale  for  these  items  was  also  developed  from  the  CIP. 

Our  findings  on  this  set  of  descriptors  include: 

• descriptive  statistics  in  line  with  expectations, 

• generally  adequate  reliability,  with  a few  exceptions. 
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• logical  correlations  with  other  education,  training,  experience,  and  licensure/certification 
descriptors, 

• particularly  high  intercorrelations, 

• some  loadings  on  the  factor  of  Education  (Descriptor  #21)  (factor  analytic  results),  and 

• some  usefulness  collectively  in  differentiating  occupations. 

These  findings  provide  mixed  evidence  for  retaining  the  descriptors  as  measured  in  this 
study.  The  descriptors  appear  to  have  reasonable  internal  validity;  however,  they  provide 
somewhat  redundant  information  between  themselves  and  with  General  Education  Level 
(Descriptor  #1). 

To  determine  if  the  same  information  could  be  obtained  with  fewer  items,  we  conducted  a 
principal  components  factor  analysis  of  these  descriptors  alone.  We  discovered  five  factors 
(technical  vocational,  business  vocational.  English/language  arts,  foreign  language,  and 
math/science).  Thus,  items  could  be  constructed  that  tap  these  broader  coursework  areas.  Use  of 
these  revised  items  would  reduce  the  overall  length  of  the  survey,  while  still  providing  essentially 
the  same  information  to  users.  However,  as  mentioned  previously,  these  findings  are  somewhat 
limited  by  the  occupations  included  in  this  sample.  Thus,  we  suggest  further  study  of  these 
descriptors  with  other  samples  of  occupations.  If  these  findings  persist,  we  recommend  reducing 
the  descriptors  down  to  a smaller  set  based  on  factor  analysis  and  the  needs  of  users. 

Instructional  Propram 

As  noted  previously.  Instructional  Program  (Descriptor  #2)  was  intended  as  a single 
descriptor  with  the  purpose  of  indicating  the  major  field  of  study  required  to  perform  the  job. 
However,  as  26%  of  the  respondents  to  the  item  indicated  that  more  than  one  instructional 
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program  was  required,  we  decided  it  may  be  useful  to  evaluate  Instructional  Program  (Descriptor 
#2)  as  42  separate  descriptors,  each  corresponding  to  the  different  instructional  programs. 

Our  findings  on  Instructional  Program  (Descriptor  #2),  examined  as  42  separate 
descriptors,  include: 

• descriptive  statistics  in  line  with  expectations,  with  many  instructional  programs  being 
endorsed  by  fewer  than  1%  of  the  sample, 

• great  variance  in  reliability  when  viewed  as  42  separate  descriptors,  with  those  programs 
receiving  little  endorsement  having  unacceptable  reliabilities,  and 

• generally  uninterpretable  correlations  with  other  descriptors. 

Our  findings  on  Instructional  Program  (Descriptor  #2),  examined  as  a single  descriptor, 
include: 

. • a profile  of  endorsement  for  the  sample  as  a whole  consistent  with  expectations, 

• acceptable  reliability,  and 

• a logical  profile  of  endorsement  for  six  example  occupations. 

Given  these  findings,  it  seems  clear  that  Instructional  Program  (Descriptor  #2)  provides 
more  than  adequate  information  for  its  intended  purpose  of  identifying  the  required  instructional 
program.  However,  given  that  many  incumbents  endorsed  more  than  one  instructional  program, 
it  may  be  prudent  to  reword  the  item  to  explicitly  allow  respondents  to  check  more  than  one. 

Even  with  this  refinement,  however,  the  item  will  provide  limited  information  in  that  it  does  not 
allow  for  the  evaluation  of  the  relevance  of  individual  instructional  programs. 

Information  on  the  relevance  of  individual  instructional  programs  may  be  valued  by 
certain  users.  For  example,  a job  seeker  with  a degree  in  Computer  Sciences  (Descriptor  #3i) 
might  want  to  examine  the  data  on  occupations  to  see  whether  or  not  a degree  in  Computer 
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Sciences  (Descriptor  #3i)  is  relevant.  To  serve  the  purpose  of  providing  information  on  the 
relevance  of  individual  instructional  programs,  42  separate  items  would  have  to  be  created,  with 
a separate  rating  for  each  one.  This  treatment  of  Instructional  Program  (Descriptor  #2)  would 
require  more  effort  on  the  part  of  the  respondents,  but  it  may  be  warranted,  depending  on  the 
needs  of  the  users.  It  is  of  note  that  the  relevance  of  individual  instructional  programs  could 
likely  also  be  assessed  by  users  through  an  examination  of  occupation  scores  within  the 
knowledge  domain. 

Our  recommendations  on  this  descriptor  are  to  (1)  reword  the  item  to  make  it  clear  that 
more  than  one  instructional  program  can  be  endorsed,  and  (2)  determine  if  the  needs  of  the  users 
to  have  information  on  the  relevance  of  individual  instructional  programs  warrants  the  costs  of 
expanding  the  item  to  42  separate  items. 

Licensure/Certification 

These  descriptors  were  meant  to  determine  (1)  whether  or  not  licensure/certification/ 
registration  is  required,  (2)  the  type  of  license/certification/registration  required,  (3)  the 
requirements  for  obtaining  licensure/certification/registration,  and  (4)  entities  requiring  the 
licensure/certification/registration.  There  are  12  items,  1 1 with  a response  scale  of  “yes/no,”  and 
one  "fill  in  the  blank."  Our  findings  on  this  set  of  descriptors  include: 

• descriptive  statistics  in  line  with  expectations, 

• generally  high  reliability, 

• logical  correlations  with  other  education,  training,  experience,  and  licensure/certification 
descriptors, 

• some  variables  have  particularly  high  intercorrelations, 

loadings  on  the  factor  of  Licensure  (factor  analytic  results)  and 

' )£ 


5-24  Anderson 


• some  usefulness  in  differentiating  occupations. 

These  findings  provide  evidence  for  retaining  the  descriptors  as  measured  in  this  study. 
The  descriptors  appear  to  have  reasonable  internal  validity,  and  they  are  useful  for  differentiating 
among  occupations. 

However,  some  of  the  descriptors  provide  statistically  redundant  information.  With  the 
occupations  in  our  sample,  some  of  the  requirements  to  obtain  and  retain  licensure 
certification/registration  have  intercorrelations  of  .90  or  higher  (i.e.,  On-the-Job  Training 
Experience  [Descriptor  #5c].  Examination  [Descriptor  #5d],  Character  References  [Descriptor 
#5e],  and  Coursework  [Descriptor  #5f]).  In  addition,  when  a non-vehicle  license  was  reported  as 
required,  it  was  also  usually  reported  as  required  by  both  the  employer  and  the  law-thus,  the 
three  descriptors  relevant  to  these  issues  had  intercorrelations  of  .90  or  higher. 

Given  these  statistical  redundancies,  the  items  might  be  collapsed  so  that,  at  least  in 
theory,  the  same  amount  of  information  could  be  gathered  with  fewer  items.  However,  we  advise 
caution  before  taking  this  step.  The  items  are  generally  distinct  in  terms  of  content,  and 
additional  samples  including  other  occupations  may  reveal  less  statistical  redundancy. 

Training/Experience 

These  descriptors  were  meant  to  determine  the  amount  of  training/experience  required  for 
the  given  occupation.  There  are  four  items  with  a response  scale  of  "years  of  experience"  based 
on  an  expanded  version  of  the  scale  used  to  assign  Specific  Vocational  Preparation  in  the  current 
DOT  (DOL,  1991).  Our  findings  on  this  set  of  descriptors  include: 

• descriptive  statistics  in  line  with  expectations, 

. • adequate  reliability. 
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• logical  correlations  with  other  education,  training,  experience,  and  licensure/certification 
descriptors, 

• loadings  on  the  factor  of  Training  (factor  analytic  results), 

• some  usefulness  in  differentiating  occupations. 

These  findings  provide  evidence  for  retaining  the  descriptors  as  measured  in  this  study. 
The  descriptors  appear  to  have  reasonable  internal  validity,  and  they  are  useful  for  differentiating 
among  occupations. 

Summary 

We  found  that  the  education,  training,  experience,  and  licensure/certification  descriptors 
have  acceptable  reliability  and  validity.  Thus,  in  general,  we  recommend  that  the  descriptor 
measures  be  retained  as  used  in  the  prototype.  A refinement  to  one  descriptor  is  strongly 
recommended  (i.e..  Instructional  Program  Requirements  [Descriptor  #2]).  Other  refinements  to 
reduce  redundancies  in  the  measures  should  be  considered  if  the  data  in  additional  samples 
indicate  the  stability  of  these  findings. 
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Figure  5-1 

0*NET  Descriptors  for  Education.  Training.  Licensure/Certification,  and  Experience.  Listed  bv 
Item  Type  and  Scale  Type 


Category 

Descriptors 

Item  Type 

Scale  Type 

General  Education  Level 

Check  level 
required 

Twelve  point  degree  level 
(1  = Less  than  a High 
School  Diploma,  12  = 
Post-Doctoral  Certificate) 

Instructional  Program 
Required  (42  descriptors): 

Agriculture 
Business/Production 
Agricultural  Sciences 
Architecture 

Area/Ethnic/Cultural  Studies 
Biological/Life  Sciences 
Business 

Management/Administrative 

Services 

Communications 

Communications 

Technologies 

Computer  Information 

Sciences 

Conservation 

Construction  Trades 

Education 

Engineering 

Engineering  Technologies 
English 

Languages/Literatures 
Foreign  Languages/Literature 
Health  Professions 
Home  Economics 
Law/legal  Studies 
Liberal  Arts/Sciences 
Library  Science 
Marketing/Distribution 
Mathematics 
Mechanics/Repairers 
Military  Technologies 
Interdisciplinary  Studies 
Parks/Recreation/Leisure/Fit 
ness 

Personal/Miscellaneous 

Services 

Philosophy/Religion 
Physical  Sciences 
Precision  Production  Trades 
Protective  Services 
Psychology 

Check  the 
instructional 
program 
required 

42  options  are  presented, 
(e.g.,  Biological/Life 
Science;  Management  & 
Admin;  Law/legal 
Studies;Transportation/M 
oving;  No  Specific  Major) 

Figure  5-1  (continued) 

Q_*NET  Descriptors  for  Education,  Training,  Licensure/Certification,  and  Experience.  Listed  bv 
Item  Type  and  Scale  Type 


Category 

Descriptors 

Item  Type 

Scale  Type 

Public 

Administration/Services 

ROTC 

Science  Technologies 
Social  Sciences/History 
Theological  Studies 
Transportation/Moving 
Visual/Perforinng  Arts 
Vocational  Home  Economics 
No  Specific  Major 

Subject  Area  Education 
Level  (15  descriptors); 

Technical  Vocational 
Business  Vocational 
English/Lang  Arts 
Oral  Communications 
Languages 
Basic  Math 
Advanced  Math 
Physical  Sciences 
Computer  Sciences 
Biological  Sciences 
Applied  Sciences 
Social  Science  s 
Arts 

Humanities 
Physical  Education 

Check  highest 
level  of 
educational 
coursework 
required 

Five  point  level  scale  (0  = 
not  required,  4 = graduate 
school  or  other  post 
undergraduate) 

Licenses  Required  (2 
descriptors) 

Commercial  Vehicle  Lie 
Non-Vehicle  License 

Check  "yes"  if 
required,  "no" 
if  not  required 

(0  = no,  1 = yes) 

Licenses  Named 

Licenses  Named 

List  licenses 
required 

Analyzed  two  ways: 
Qualitative,  and 
Quantitative  (0  = no 
license  listed;  1 = one  or 
more  licenses  listed) 

Requirement  to  Obtain  a 
License  License  (6 
descriptors) 

Post  Secondary  Degree 
Graduate  Degree 
On-the-Job  Training 
Examination 
Character  References 
Coursework 

Check  "yes"  if 
required,  "no" 
if  not  required 

(0  = no,  1 = yes) 

Who  Requires  License  (3 
descriptors) 

Law 

Employer 

Union/Association 

Check  "yes"  if 
required  by 
this -entity, 

"no"  if  not 

(0  = no,  1 = yes) 

Figure  5-1  (continued) 

Q*NET  Descriptors  for  Education.  Training.  Licensure/Certification,  and  Experience.  Listed  bv 
Item  Type  and  Scale  Type 


Category 

Descriptors 

Item  Type 

Scale  Type 

required 

Related  Work  Experience  (4 
descriptors) 

On-Site  Training  Experience 
On-the-Job  Training 
Experience 

Apprenticeship  Experience 

Check  level  of 

experience 

required 

Eleven  point  scale  (0  = not 
applicable  or  none,  10  = 
over  10  years) 

ERiC 


Figure  5-2 

Thirty-Four  Occupations  With  Four  or  More  Incumbents  Completing  the  Skills  Questionnaire 


Occupation  Code 

Occupation  Title 

Number  of  Respondents 

15005 

Education  Administrators 

ii  i 

19005 

General  Managers  & Top  Executives 

38 

22135 

Mechanical  Engineers 

6 

25105 

Computer  Programmers 

6 

27311 

Recreation  Workers 

7 

31303 

Teachers,  Preschool 

4 

31305 

Teachers,  Elementary  School 

7 

31502 

Librarians,  Professional 

4 

32502 

Registered  Nurses 

29 

32902 

Medical  & Clinical  Laboratory  Technologists 

8 

49008 

Salespersons,  Except  Scientific  & Retail 

7 

49011 

Salespersons,  Retail 

22 

49017 

Counter  & Rental  Clerks 

5 

49021 

Stock  Clerks,  Sales  Floor 

8 

49023 

Cashiers 

27 

Tellers 

4 

51002 

First  Line  Supervisors,  Clerical/Administrative 

57 

53121 

Loan  & Credit  Clerks 

4 

53311 

Insurance  Claims  Clerks 

ii 

53905 

Teachers’ Aides  & Assistants,  Clerical 

8 

55108 

Secretaries,  Except  Legal  & Medical 

66  i 

55305 

Receptionists  & Information  Clerks 

6 

55338 

Bookkeeping,  Accounting,  & Auditing  Clerks 

23 

55347 

General  Office  Clerks 

68 

61005 

Police  & Detective  Supervisors 

11 

63014 

Police  Patrol  Officers 

17 

65008 

Waiters  & Waitresses 

15 

Cooks , Restaurant 

4 

65038 

Food  Preparation  Workers 

14 

66008 

Nursing  Aides,  Orderlies,  & Attendants 

14 

67005 

Janitors  & Cleaners 

29 

Other  Machinery 

4- 

85132 

Maintenance  Repairers,  General  Utility 

36 

87902 

Earth  Drillers,  Except  Oil  & Gas 

8 

89108 

Machinists 

4 

92974 

Packaging  & Filling  Machine  Operators 

10 

97102 

Truck  Drivers,  Heavy  or  Tractor  Trailer 

9 

97111 

Bus  Drivers,  Schools 

10 

4 


10 


Table  5-1 

Descriptive  Statistics  Across  Occupations  and  Reliability  Estimates  for  Rated  Differences 
Between  Occupations:  Education.  Training.  Licensure,  and  Experience 


Variable 


Descriptor 

M 

SD 

SEMa 

Skb 

1.  General  Education  Level 

2.  Instructional  Program 

3.43 

2.01 

.37 

.97 

2a. 

Agriculture  Business/Production 

.01 

.05 

;03 

.59 

2b. 

Agricultural  Sciences 

.01 

.05 

.04 

.43 

2c.  . 

Architecture 

.00 

.02 

.01 

.52 

2d. 

Area/Ethnic/Cultural  Studies 

.02 

.05 

.03 

.57 

2e. 

Biological/Life  Sciences 

.03 

.07 

.04 

.75 

2f.  Business  Management/ 

Administrative  Services 

.12 

.15 

.08 

.74 

2g- 

Communications 

.10 

.11 

.09 

.30 

2h. 

Communications  Technologies 

.01 

.03 

.03 

.00 

2i. 

Computer  Information  Sciences 

.12 

.18 

.11 

.66 

2j- 

Conservation 

.00 

.02 

.02 

.22 

2k. 

Construction  Trades 

.03 

.10 

.04 

.81 

21. 

Education 

.10 

.15 

.09 

.64 

2m. 

Engineering 

.03 

.08 

.05 

.63 

2n. 

Engineering  Technologies 

.03 

.10 

.05 

.76 

2o. 

English  Languages/Literatures 

.07 

.09 

.08 

.25 

2p. 

Foreign  Languages/Literature 

.03 

.09 

.06 

.59 

2q. 

Health  Professions 

.11 

.23 

.05 

.96 

2r. 

Home  Economics 

.01 

.02 

.02 

.43 

2s. 

Law/legal  Studies 

.05 

.17 

.03 

.96 

2t. 

Liberal  Arts/Sciences 

.01 

.02 

.02 

.08 

2u. 

Library  Science 

.03 

.17 

.03 

.97 

2v. 

Marketing/Distribution 

.04 

.08 

.06 

.47 

2w. 

Mathematics 

.11 

.13 

.09 

•51 

2x. 

Mechanics/Repairers 

.08 

.17 

.05 

.91 

2y. 

Military  Technologies 

.01 

.02 

.02 

.46 

2z. 

Interdisciplinary  Studies 

.02 

.04 

.03 

.41 

2aa. 

Parks/Recreation/Leisure/Fitness 

.00 

.02 

.02 

.22 

2bb. 

Personal/Miscellaneous  Services 

.02 

.05 

.03 

.51 

2cc. 

Philosophy/Religion 

.01 

.03 

.03 

.08 

2dd. 

Physical  Sciences 

.01 

.03 

.03 

.30 

2ee. 

Precision  Production  Trades 

.01 

.05 

.02 

.73 

2ff. 

Protective  Services 

.02 

.04 

.03 

.51 

2gg- 

Psychology 

.04 

.06 

.05 

.38 

2hh. 

Public  Administration/Services 

.03 

.06 

.05 

.32 

Table  5-1  (continued) 

Descriptive  Statistics  Across  Occupations  and  Reliability  Estimates  for  Rated  Differences 
Between  Occupations:  Education.  Training.  Licensure,  and  Experience 


Variable 


Descriptor 

M 

SD 

SEM 

Ik 

2ii. 

ROTC 

.00 

.02 

.02 

.34 

2jj- 

Science  Technologies 

.01 

.02 

.02 

.00 

2kk. 

Social  Sciences/History 

.01 

.02 

.03 

.00 

211. 

Theological  Studies 

.01 

.05 

.02 

.73 

2mm. 

Transportation/Moving 

.04 

.11 

.05 

.77 

2nn. 

Visual/Performing  Arts 

.01 

.03 

.02 

.44 

2oo. 

Vocational  Home  Economics 

.03 

.07 

.04 

.75 

2pp.  No  Specific  Major 
3.  Subject  Area  Education  Level 

.43 

.26 

.12 

.81 

3a. 

Technical  Vocational 

.69 

.44 

.24 

.71 

3b. 

Business  Vocational 

1.08 

.41 

.23 

.67 

3c. 

English/Language  Arts 

1.55 

.68 

.31 

.79 

3d. 

Oral  Communications 

1.61 

.67 

.30 

.80 

3e. 

Languages 

.99 

.46 

.41 

.20 

3f. 

Basic  Math 

1.40 

.56 

.24 

.81 

3g. 

Advanced  Math 

1.28 

.78 

.36 

.78 

3h. 

Physical  Sciences 

1.19 

.77 

.38 

.76 

3i. 

Computer  Sciences 

1.36 

.67 

.35 

.73 

3j. 

Biological  Sciences 

1.15 

.75 

.35 

.78 

3k. 

Applied  Sciences 

1.31 

.71 

.37 

.73 

31. 

Social  Sciences 

1.29 

.77 

.34 

.80 

3m. 

Arts 

.81 

.52 

.37 

.49 

3n. 

Humanities 

1.06 

.70 

.39 

.68 

3o,  Physical  Education 
4.  Licenses 

.71 

.47 

.32 

.53 

4a. 

Commercial  Vehicle  License 

.15 

.28 

.07 

.93 

4b. 

Non- Vehicle  License 

.29 

.33 

.08 

.94 

4c.  Licenses  Named 
5.  Requirements  to  Obtain  a License 

.27 

.33 

.07 

.95 

5a. 

Post  Secondary  Degree 

.13 

.26 

.05 

.96 

5b. 

Graduate  Degree 

.04 

.06 

.05 

.50 

5c. 

On-the-Job  Training 

.24 

.30 

.08 

.93 

5d. 

Examination 

.25 

.31 

.07 

.95 

5e. 

Character  References 

.17 

.24 

.08 

.89 

5f. 

Coursework 

.20 

.28 

.07 

.93 

Table  5-1  (continued) 

Descriptive  Statistics  Across  Occupations  and  Reliability  Estimates  for  Rated  Differences 
Between  Occupations:  Education.  Training.  Licensure,  and  Experience 


Variable 


Descriptor 

M 

SD 

SEM 

Dc 

6.  Who  Requires  License 

6a.  Law 

.28 

.33 

.08 

.94 

6b.  Employer 

.34 

.36 

.09 

.94 

6c.  Union/Association 

.17 

.22 

.07 

.90 

7.  Experience 

7a.  Related  Work  Experience 

3.80 

1.69 

.64 

.86 

7b.  On-Site  Training  Experience 

1.76 

.85 

.54 

.59 

7c.  On-the-Job  Training  Experience 

2.49 

.97 

.58 

.65 

7d.  Apprenticeship  Experience 

.81 

.79 

.39 

.75 

Note.  Statistics  are  based  on  34  occupations  with  Education,  Training, 

Licensure,  and  Experience  questionnaire  responses  from  at  least  4 incumbents 
(mean  number  of  incumbents  = 17.50,  median  =11,  harmonic  mean  = 9.04). 

aThis  estimate  of  the  standard  error  of  measurement  was  calculated  as  SEM  = SD*  -^ ( 1 - rv) . 

'’This  estimate  of  reliability  was  obtained  by  calculating  the  intraclass  correlation  fork  ratings 
across  occupations:  ICC(l,k)  = fBMS-WMSl/BMS  (Shrout  & Reiss,  1979),  where  k is  the 
harmonic  mean  of  the  number  of  ratings  provided  on  each  occupation. 


Table  5-2 

Reliability  of  Rated  Differences  Between  Occupations  Considering  Varying  Numbers  nf  Batprc- 
Education,  Training,  Licensure  and  Experience 


Variable 


Descriptor 

Iia 

l30b 

1.  General  Education  Level 

76 

99 

2.  Instructional  Program 

2a. 

Agriculture  Business/Production 

14 

83 

2b. 

Agricultural  Sciences 

08 

71 

2c. 

Architecture 

11 

79 

2d. 

Area/Ethnic/Cultural  Studies 

13 

82 

2e. 

Biological/Life  Sciences 

25 

91 

2f. 

Business  Management/ 

24 

90 

Administrative  Services 

2g- 

Communications 

05 

59 

2h. 

Communications  Technologies 

00 

00 

2i. 

Computer  Information  Sciences 

18 

87 

2j- 

Conservation 

03 

48 

2k. 

Construction  Trades 

32 

94 

21. 

Education 

16 

85 

2m. 

Engineering 

16 

85 

2n. 

Engineering  Technologies 

26 

91 

2o. 

English  Languages/Literatures 

03 

52 

2p. 

Foreign  Languages/Literatures 

14 

83 

2q. 

Health  Professions 

73 

99 

2r. 

Home  Economics 

08 

72 

2s. 

Law/legal  Studies 

74 

99 

2t. 

Liberal  Arts/Sciences 

01 

22 

2u. 

Library  Science 

80 

99 

2v. 

Marketing/Distribution 

09 

75 

2w. 

Mathematics 

10 

77 

2x. 

Mechanics/Repairers 

52 

97 

2y. 

Military  Technologies 

09 

74 

2z. 

Interdisciplinary  Studies 

07 

70 

2aa. 

Parks/Recreation/Leisure/Fitness 

03 

48 

2bb. 

Personal/Miscellaneous  Services 

10 

77 

2cc. 

Philosophy/Religion 

01 

21 

2dd. 

Physical  Sciences 

04 

58 

2ee. 

Precision  Production  Trades 

23  f 

90 

2ff. 

Protective  Services 

10 

78 

2gg- 

Psychology 

06 

67 

2hh. 

Public  Administration/Services 

05 

61 

2ii. 

ROTC 

05 

64 

2jj. 

Science  Technologies 

00 

00 

2kk. 

Social  Sciences/History 

00 

00 

ft 

i s 1 \ 

t 

<t  *. 
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Table  5-2  (continued) 

Reliability  of  Rated  Differences  Between  Occupations  Considering  Varying  Numbers  of  Raters: 
Education.  Training,  Licensure  and  Experience 


Variable 


Descriptor 

Ei 

£30 

211. 

Theological  Studies 

23 

90 

2mm. 

Transportation/Moving 

27 

92 

2nn. 

Visual/Performing  Arts 

08 

73 

2oo. 

Vocational  Home  Economics 

25 

91 

2pp.  No  Specific  Major 
3.  Educational  Subject  Area 

31 

93 

3a. 

Technical  Vocational 

22 

89 

3b. 

Business  Vocational 

18 

87 

3c. 

English/Language  Arts 

29 

93 

3d. 

Oral  Communications 

31 

93 

3e. 

Languages 

03 

45 

3f. 

Basic  Math 

33 

94 

3g- 

Advanced  Math 

28 

92 

3h: 

Physical  Sciences 

26 

91 

3i. 

Computer  Sciences 

23 

90 

3j. 

Biological  Sciences 

28 

92 

3k. 

Applied  Sciences 

23 

90 

31. 

Social  Sciences 

31 

93 

3m. 

Arts 

09 

76 

3n. 

Humanities 

19 

88 

3o.  Physical  Education 

4.  Licenses 

11 

79 

4a. 

Commercial  Vehicle  License 

60 

98 

4b. 

Non- Vehicle  License 

65 

98 

4c.  Licenses  Named 

5.  Requirements  to  Obtain  a License 

67 

98 

5a. 

Post  Secondary  Degree 

71 

99 

5b. 

Graduate  Degree 

10 

77 

5c. 

On-the-Job  Training 

61 

98 

5d. 

Examination 

66 

98 

5e. 

Character  References 

48 

96 

5f.  Coursework 

6.  Who  Requires  License 

61 

98 

6a. 

Law 

62 

98 

6b. 

Employer 

64 

98 

6c. 

Union/Association 

51 

97 

Table  5-2  (continued) 

Reliability  of  Rated  Differences  Between  Occupations  Considering  Varying  Numherc  r>f  Rat^rc- 
Education,  Training.  Licensure  and  Experience 

Variable 

Descriptor ij r30 

7.  Experience 


7a. 

Related  Work  Experience 

40 

95 

7b. 

On-site  Training  Experience 

14 

83 

7c. 

On-the-Job  Training  Experience 

17 

86 

7d. 

Apprenticeship  Experience 

25 

91 

Note.  Reliability  estimates  are  based  on  34  occupations  with  Education,  Training,  Licensure,  and 
Experience  questionnaire  responses  from  at  least  4 incumbents  (mean  number  of  incumbents  = 
17.50,  median  =11,  harmonic  mean  = 9.04).  Decimals  are  omitted. 

“Single  rater  estimates  of  reliability  were  obtained  by  calculating  the  intraclass  correlation  for 
single  judge  ratings  across  occupations:  ICC(1,1)  = rBMS-WMSl/rBMS+Oc-  1YWMS1  fShrout 
& Fleiss,  1979),  where  k is  the  harmonic  mean  of  the  number  of  ratings  provided  on  each 
occupation. 

‘’Estimates  of  reliability  for  30  raters  were  obtained  by  applying  the  Spearman-Brown  correction 
formula  to  the  single  rater  reliability  estimates. 


1 


416 


Table  5-4a 

Analysis  of  Variance  for  Descriptor,  Occupation,  and  Relevant  Interactions  as  Sources  of 
Variation:  Instructional  Program  Required 


Source  of  Variation 

SS 

df 

MS 

F 

Occupations 

8.21 

33 

0.25 

2.02* 

S(Occupations) 

70.14 

564 

0.12 

Descriptor 

62.04 

41 

1.51 

54.26* 

Descriptor  x Occupations 

171.62 

1353 

0.13 

4.55* 

Descriptor  x S (Occupations) 

644.96 

23124 

0.03 

Note.  Occupations  are  treated  as  random,  between-subjects  effects,  while  descriptors  are  treated 
as  fixed,  within-subjects  effects. 

*E  < .05 


Table  5-4b 


ruimyoio  v axioms  i ui 

Variation:  Educational  Subiect  Area 

uiu u*  ana  ^eievam 

. interacuons  as  so 

>urces  or 

Source  of  Variation 

SS 

df 

MS 

F 

Occupations 

1407.89 

33 

42.66 

3.65* 

S(Occupations) 

6591.91 

564 

11.69 

Descriptor 

334.66 

14 

23.90 

43.63* 

Descriptor  x Occupations 

968.52 

462 

2.10 

3.83* 

Descriptor  x S (Occupations) 

4326.42 

7896 

0.55 

Note.  Occupations  are  treated  as  random*  between-subjects  effects,  while  descriptors  are  treated 
as  fixed,  within-subjects  effects. 

*E  < .05 


Table  5 -4c 

Analysis  of  Variance  for  Descriptor,  Occupation,  and  Relevant  Interactions  as  Sources  of 
Variation:  Licensure 


Source  of  Variation 

SS 

df 

MS 

F 

Occupations 

355.08 

33 

10.76 

27.83* 

S (Occupations) 

218.09 

564 

0.39 

Descriptor 

23.85 

11 

2.17 

44.35* 

Descriptor  x Occupations 

116.64 

363 

0.32 

6.57* 

Descriptor  x S(Occupations) 

303.31 

6204 

0.05 

Note.  Occupations  are  treated  as  random,  between-subjects  effects,  while  descriptors  are  treated 
as  fixed,  within-subjects  effects. 

*P  < .05 


Table  5-4d 

Analysis  of  Variance  for  Descriptor  Occupation  and  Relevant  Interactions  as  Sources  of  Variation: 
Experience 


Source  of  V ariation 

SS 

df 

MS 

F 

Occupations 

1277.29 

33 

38.71 

3.97* 

S(Occupations) 

5503.88 

564 

9.76 

Descriptor 

1465.47 

3 

488.49 

172.33* 

Descriptor*  Occupations 

1357.96 

99 

13.72 

4.84* 

Descriptors  (Occupations) 

4796.17 

1692 

2.83 

Note.  Occupations  are  treated  as  random,  between-subjects  effects,  while  descriptors  are  treated 
as  fixed,  within-subjects  effects. 

*£>  < .05 


Table  5-5 

Interater  Agreement  Coefficients  for  Each  Scale  Type:  Education,  Training.  Licensure,  and  Experience- 


Scale  Type 

Number  of  Raters  on  Each  Variable 

Ik 

Ii 

DO 

Instructional  Program 

78 

28 

92 

Educational  Subject  Area 

74 

24 

90 

Licensure 

85 

38 

95 

Experience 

79 

30 

93 

Note.  Interrater  agreement  coefficient  estimates  are  based  on  34  occupations  with  Education, 
Training,  Licensure,  and  Experience  questionnaire  responses  from  at  least  4 incumbents  (mean 
number  of  incumbents  = 17.50,  median  =11,  harmonic  mean  = 9.04).  Full  sample  interrater 
agreement  coefficients  (riO  were  obtained  by  considering  the  "Descriptor  x Occupations"  term 
from  Tables  5-4a,  5-4b,  5 -4c,  and  5-4d  as  true  variance.  Error  variance  was  defined  as  the 
"Descriptor  x S(Occupations)"  term.  Estimates  of  reliability  for  1 and  30  raters  were  obtained  by 
applying  the  Spearman-Brown  correction  formula  to  the  k rater  reliability  estimates,  where  k is 
the  harmonic  mean  of  the  number  of  raters  for  each  occupation.  Decimals  are  omitted. 


Table  5-9 

Intercorrelations  of  Descriptors  (Occupation-Level  Data):  Education,  Training,  Licensure,  and  Experience 
Descriptor  1 2a  2b  2c  2d  2e  2f  2e  2h  2i  2i  2k 
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Table  5-9  (continued) 

Intercorrelations  of  Descriptors  (Occupation-Level  Data):  Education.  Training.  Licensure,  and  Experience 
Descriptor 1 2a  2b  2c  2d  2e  2f  2g  2h  2i  2i  2k 
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Table  5-9  (continued) 

Intercorrelations  of  Descriptors  (Occupation-Level  Data):  Education.  Training.  Licensure,  and  Experience 
Descriptor  2jj  2kk  211  2mm  2nn  2oo  2pp  3a  3b  3c  3d  3e 
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Table  5-9  (continued) 

Intercorrelations  of  Descriptors  (Occupation-Level  Data):  Education.  Training.  Licensure,  and  Experience 

Descriptor 31  3m  3n  3o  4a  4b  4c  5a  5b  5c  5d  5e 
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Table  5-ld 

Intercorrelations  of  Descriptors  (Individual-Level  Data):  Education.  Training.  Licensure,  and  Experience 
DescHptor 1 2a  2b  2c  2d  2e  2f  2g  2h  2i  2j  2k 
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Table  5- Id  (continued) 

Intercorrelations  of  Descriptors  (Individual-Level  Data):  Education.  Training.  Licensure,  and  Experience 
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Table  5-10  (continued) 

Intercorrelations  of  Descriptors  (Individual-Level  Data!:  Education.  Training.  Licensure,  and  Experience 
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Table  5-10  (continued) 

Intercorrelations  of  Descriptors  (Individual -Level  Data):  Education.  Training.  Licensure,  and  Experience 
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Table  5-10  (continued) 

Intercorrelations  of  Descriptors  (Individual-Level  Data):  Education.  Training.  Licensure,  and  Experience 
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Table  5-11 

Principal  Components  Analysis  Pattern  Matrix  for  Education  Training.  Licensure/Ce.rtifipatinn 
and  Experience 


Factor 


Descriptor 

FI 

F2 

F3 

Communality 

1.  General  Level  of  Education 

.87 

.08 

-.19 

.79 

3.  Subject  Area  Education  Level 

3a. 

Technical  Vocational 

.34 

.13 

.68 

.59 

3b. 

Business  Vocational 

.32 

-.55 

.27 

.48 

3c. 

English/Language  Arts 

.92 

.07 

-.03 

.86 

3d. 

Oral  Communications 

.93 

-.01 

.02 

.87 

3e. 

Languages 

.71 

-.18 

.15 

.56 

3f. 

Basic  Math 

.90 

.02 

-.02 

.81 

3g. 

Advanced  Math 

.93 

.01 

.03 

.87 

3h. 

Physical  Sciences 

.91 

.25 

.05 

.88 

3i. 

Computer  Sciences 

.84 

-.33 

.04 

.81 

3j- 

Biological  Sciences 

.86 

.32 

-.10 

.86 

3k. 

Applied  Sciences 

.86 

.25 

.10 

.81 

31. 

Social  Sciences 

.92 

.26 

-.04 

.91 

3m. 

Arts 

.79 

.00 

.00 

.63 

3n. 

Humanities 

.84 

.14 

-.01 

.72 

3o. 

Physical  Education 

.72 

.19 

.19 

.60 

4.  Licenses 

4a. 

Commercial  Vehicle  License 

-.28 

.40 

.16 

.26 

4b. 

Non-Vehicle  License 

.17 

.97 

.04 

.97 

4c. 

Licenses  Named 

.17 

.96 

.02 

.96 

5.  Requirements  to  Obtain  a License 

5a. 

Post  Secondary  Degree 

.48 

.67 

-.17 

.71 

5b. 

Graduate  Degree 

.35 

.41 

.00 

.29 

5c. 

On-the-Job  Training 

.11 

.97 

.04 

.95 

5d. 

Examination 

.11 

.96 

.04 

.93 

5e. 

Character  References 

.08 

.92 

-.02 

.84 

5f. 

Coursework 

.16 

.95 

-.01 

.92 

6.  Who  Requires  License 

6a. 

Law 

-.06 

.95 

.11 

.92 

6b. 

Employer 

.11 

.94 

.05 

.90 

6c. 

Union/Association 

.15 

.86 

.02 

.75 

7.  Experience 

7a. 

Related  Work  Experience 

.34 

-.42 

.29 

.38 

7b. 

On-Site  Training  Experience 

-.01 

-.06 

.79 

.63 

7c. 

On-the-Job  Training  Experience 

-.13 

.05 

.82 

.69 

Id. 

Apprenticeship  Experience 

-.07 

.02 

.85 

.72 

Percent  of  Variance 

40 

26 

9 

Eigenvalues 

11.28 

9.76 

2.82 

O 

ERIC 


460 


Table  5-1 1 (continued) 

Principal  Components  Analysis  Pattern  Matrix  Education  Training.  Licensure/Certification,  and 
Experience 


Note.  N = 34.  The  correlation  matrix  was  based  on  means  calculated  at  the  occupation  level.  FI 
Education,  F2  = Licensure,  F3  = Training.  These  loadings  are  based  on  a varimax  orthogonal 
rotation. 


Table  5-12 

Descriptor  Means  and  Standard  Deviations  on  Six  Example  Occupations:  Education.  Training.  Licensure,  and  Experience 
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Table  5-12  (continued) 

Descriptor  Means  and  Standard  Deviations  on  Six  Example  Occupations:  Education.  Training.  Licensure,  and  Experience 

Occupations . 
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T able  5-12  (continued) 

Descriptor  Means  and  Standard  Deviations  on  Six  Example  Occupations:  Education.  Training.  Licensure,  and  Experience 
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Table  5-12  (continued) 

Descriptor  Means  and  Standard  Deviations  on  Six  Example  Occupations:  Education.  Training.  Licensure,  and  Experience 
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Table  5-13 

Rotated  Correlations  Between  Discriminating  Variables  and  Canonical  Discriminant  Functions: 
Education,  Training.  Experience,  and  Licensure 


Functions 


Descriptor 

FI 

F2 

F3 

F4 

F5 

F6 

IT2 

2 

T1 

1.  General  Education  Level 
3.  Subject  Area  Education  Level 

.11 

.82 

.06 

.08 

.26 

.19 

.79 

.64 

3a. 

Technical  Vocational 

.01 

-.21 

.04 

.01 

.18 

.20 

.12 

.17 

3b. 

Business  Vocational 

-.08 

.09 

.00 

.04 

.09 

.11 

.04 

.15 

3c. 

English/Language  Arts 

.07 

.09 

-.05 

-.04 

.23 

.18 

.10 

.22 

3d. 

Oral  Communications 

.00 

.21 

-.03 

.02 

.30 

.09 

.14 

.23 

3e. 

Languages 

-.08 

-.03 

.05 

.07 

.11 

-.05 

.03 

.07 

3f. 

Basic  Math 

-.06 

.06 

.04 

.01 

.61 

.00 

.39 

.24 

3g- 

Advanced  Math 

.02 

.07 

.05 

.00 

.44 

-.03 

.20 

.21 

3h. 

Physical  Sciences 

.04 

-.06 

.20 

.05 

.25 

.01 

.11 

.20 

3i. 

Computer  Sciences 

-.02 

.09 

.01 

-.03 

.13 

.05 

.03 

.18 

3j. 

Biological  Sciences 

.09 

.05 

.29 

.01 

.02 

.01 

.10 

.21 

3k. 

Applied  Sciences 

.09 

.07 

.13 

-.02 

.18 

.05 

.07 

.18 

31. 

Social  Sciences 

.10 

.14 

.04 

-.08 

.17 

.05 

.07 

.23 

3m. 

Arts 

.02 

.04 

.05 

.02 

.02 

-.04 

.01 

.10 

3n. 

Humanities 

.04 

.11 

-.01 

-.03 

-.01 

.02 

.01 

.15 

3o.  Physical  Education 
4.  Licenses 

.04 

-.04 

-.09 

-.04 

.13 

-.03 

.03 

.11 

4a. 

Commercial  Vehicle  Lie 

.14 

.05 

-.03 

.90 

.01 

.04 

.84 

.46 

4b. 

Non-Vehicle  License 

.73 

.03 

.07 

.03 

-.01 

.00 

.54 

.51 

4c.  Licenses  Named 
5.  Requirements  to  Obtain  a License 

.76 

.04 

.10 

.01 

-.05 

-.05 

.60 

.53 

5a. 

Post  Secondary  Degree 

.45 

.06 

.55 

-.12 

.20 

.03 

.57 

.57 

5b. 

Graduate  Degree 

.16 

-.05 

.08 

.04 

.03 

.00 

.04 

.10 

5c. 

On  the  Job  Training 

.64 

-.04 

.06 

.05 

.05 

-.09 

.43 

.47 

5d. 

Examination 

.73 

.03 

.07 

.01 

-.03 

.09 

.54 

.52 

5e. 

Character  References 

.44 

.05 

.01 

.03 

-.04 

-.10 

.20 

.35 

5f.  Coursework 
6.  Who  Requires  License 

.61 

.12 

-.03 

-.03 

-.11 

-.09 

.41 

.47 

6a. 

Law 

.66 

-.03 

-.05 

.08 

.08 

.03 

.45 

.48 

6b. 

Employer 

.65 

.00 

-.05 

.02 

.11 

.11 

.45 

.50 

6c. 

Union/Assoc 

.23 

.00 

.07 

.05 

.05 

.04 

.07 

.38 

Table  5-13  (continued) 


Education,  Training.  Experience,  and  Licensure 

Functions 

Descriptor 

FI 

F2 

F3 

F4 

F5 

F6 

rr2 

„2 

T| 

7.  Experience 

7 a.  Related  Work  Exp 

-.09 

.15 

.02 

.04 

.00 

.81 

.69 

.29 

7b.  On-site  Training 

.10 

.04 

-.10 

-.06 

.06 

.14 

.05 

.13 

7c.  On  the  Job  Training 

.04 

-.02 

.09 

.05 

.01 

.07 

.02 

.14 

7d.  Apprenticeship 

-.02 

.07 

-.03 

.04 

.01 

.01 

.01 

.19 

Rc 

.87 

.83 

.70 

.63 

.57 

.53 

Percent  of  Variance 

40 

23 

10 

7 

5 

4 

Eigenvalue 

2.98 

2.18 

.95 

.66 

.49 

.39 

Note.  Statistics  are  based  on  34  occupations  with  responses  from  at  least  4 incumbents  (mean 
number  of  incumbents  = 17.50,  median  =11,  harmonic  mean  = 9.04).  FI  = License  required,  F2 
= Education  Level,  F3  = Degree  Required  to  Obtain  Licensure,  F4  = Commercial  Vehicle 
License,  F5  = Coursework  Required,  and  F6  = Work  Experience. 

FF  = Sum  of  squared  rotated  standardized  discriminant  function  coefficients  across  6 functions. 
T)2  = Variance  in  ratings  accounted  for  by  occupations. 

The  statisitcs  “Rc,”  “Percent  of  Variance,”  and  “Eigenvalue,”  were  calculated  based  on  the 
unrotated  discriminant  functions. 


Chapter  6 

Generalized  Work  Activities: 

Evidence  for  the  Reliability  and  Validity  of  the  Measures 

Walter  C.  Borman 

Personnel  Decisions  Research  Institutes,  Incorporated 
P.  Richard  Jeanneret 
Jeanneret  and  Associates,  Incorporated 
U.  Christean  Kubisiak 
and 

Mary  Ann  Hanson 

Personnel  Decisions  Research  Institutes,  Incorporated 


Historically,  the  focus  of  job  analysis  has  been  on  the  development  of  rather  specific 
information  that  documents  either  job  content  (e.g.,  tasks)  or  job  requirements  (e.g.,  skills). 

While  a high  degree  of  specificity  can  be  extremely  useful  for  many  practical  applications  (e.g., 
design  of  skill  training),  for  many  other  important  purposes  (e.g.,  determining  the  similarities  in 
jobs  from  different  occupational  settings),  it  may  not  be  as  helpful.  Thus,  the  degree  of  generality 
in  a job  analytic  process  should  be  determined  by  both  organizational  needs  (i.e.,  what  use  will 
be  made  of  the  job  analysis  information  obtained  by  the  process),  as  well  as  the  nature  of 
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predictive  hypotheses  that  will  be  derived  (i.e.,  documenting  the  job-relatedness  of  a construct 
measured  by  a selection  procedure). 

There  is  one  other  variable  that  has  considerable  influence,  namely  the  type  of  descriptor 
used  in  the  job  analytic  process.  McCormick  (1979)  discussed  at  length  the  nature  of  job 
descriptors,  and  classified  them  into  two  broad  categories:  job-oriented  and  worker-oriented.  Job- 
oriented  descriptors  are  best  characterized  by  task  statements  which  are  specific  to  a particular 
job;  worker-oriented  descriptors  on  the  other  hand  describe  or  imply  the  human  behaviors  that 
are  involved  in  accomplishing  work.  There  has  been  little  research  on  the  efficacy  of  the  two 
types  of  descriptors,  although  the  research  that  has  been  conducted  indicates  that  one  may  well 
reach  different  conclusions  depending  on  the  type  of  descriptor  used.  What  is  clear  however  is 
that  when  examining  relationships  across  occupations  the  most  reasonable  bridges  can  be 
established  with  generalizable  work  behaviors.  (See  for  example,  Ballentine,  Cunningham,  & 
Wimpee,  1992;  Colbert  & Taylor,  1978;  Cornelius,  Cairon,  & Collins,  1979;  DeNisi  & 
McCormick,  1974;  Dowell  & Wexley,  1978;  Hamer  & Cunningham,  1981;  McCormick,  DeNisi, 
& Shaw,  1977;  McCormick  & Jeanneret,  1988;  Pearlman,  1980;  Sackett,  Cornelius,  & Carron, 
1981,  Taylor,  1978,  and  Taylor  & Colbert,  1978.)  Consequently,  as  the  development  of  work 
teams  continues  to  emerge,  as  work  is  designed  into  larger  blocks  of  activity,  as  placement  and 
career  pathing  decisions  become  more  prominent,  and  as  determining  the  requirements  for  work 
becomes  more  critical  to  preparing  tomorrow’s  workforce,  the  measurement  of  work  using 
Generalized  Work  Activities  (GW As)  will  be  an  important  component  of  the  0*NET  content 
model.  Whether  used  in  a standalone  maimer  or  in  combination  with  other  job  analysis  measures, 
it  is  intended  that  GW  As  will  be  an  important  information  source  when  matching  people  to  work. 
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The  criteria  that  were  used  to  determine  what  constructs  would  qualify  as  GW  As  were  as 
follows: 

• Constructs  must  be  broad  in  scope  with  applicability  to  a wide  range  of  occupations. 

• Constructs  must  be  based  on  job  analytic  research. 

• Constructs  must  be  characteristics  of  the  underlying  structure  of  work. 

Based  on  these  criteria,  a simple  definition  of  a GWA  emerged: 

A Generalized  Work  Activity  is  an  aggregation  of  similar  job  activities/behaviors  that 
underlies  the  accomplishment  of  major  work  functions. 

Further  information  on  the  evolution  of  GW  As  and  particularly  their  importance  to 
understanding  the  structure  of  work  is  provided  in  the  following  section  of  this  chapter. 

Background 

The  origin  of  the  GWA  concept  can  be  traced  to  research  in  the  1960s  by  Ernest 
McCormick  and  his  students  (Cunningham  & McCormick,  1964;  Gordon  & McCormick,  1963; 
Jeanneret  & McCormick,  1969;  Mecham  & McCormick,  1969;  and  McCormick,  Mecham,  & 
Jeanneret  1972).  Conceptually,  McCormick  and  his  associates  hypothesized  and  subsequently 
substantiated  that  GW  As  (or  what  McCormick  labeled  as  job  dimensions)  could  serve  as  the 
linking  pins  for  generalizing  the  results  from  empirical  selection  test  validation  studies  to 
situations  where  sample  sizes  were  small  or  other  validation  strategies  were  not  feasible. 

What  emanated  from  the  initial  research  was  a methodology  that  integrated  the  strengths 
and  capabilities  of  both  job-oriented  (task-based)  and  worker-oriented  (requirements-based)  job 
analytic  techniques.  The  original  job  analysis  data  collection  instrument  designed  to  measure 
worker-oriented  variables  was  known  as  the  Work  Activity  Profile  (WAP;  McCormick,  1964). 
Two  instruments -subsequently -emerged  that  have  been  used  inmnnerous  job  analysis  studies— 
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the  Position  Analysis  Questionnaire  (PAQ)  by  McCormick,  Jeanneret,  and  Mecham  (1969, 1972) 
and  the  Generalized  Work  Inventory  (GWI)  by  Cunningham  and  Ballentine  (1982).  Initial  studies 
conducted  with  these  two  job  analysis  questionnaires  were  designed  to  obtain  information  on 
large,  diverse  samples  of  jobs  encompassing  numerous  occupational  domains.  Importantly,  this 
information  was  also  used  to  conduct  factor  analyses  in  an  attempt  to  identify  the  underlying 
structure  of  work.  Results  of  the  factor  analyses  were  reasonably  consistent  beginning  with  the 
pioneering  efforts  of  Jeanneret  (1969)  using  data  based  on  the  PAQ,  through  replications  by 
Harvey  (1987)  and  then  to  subsequent  follow-on  studies  with  very  large  databases  by 
McCormick,  Jeanneret,  and  Mecham  (1989)  and  Jeanneret  (1990).  The  same  set  of  job 
dimensions  (factors)  repeatedly  appeared.  Further,  the  same,  or  very  similar  factors  were  being 
identified  by  others  using  related  instruments:  the  GWI  and  its  longer  version-the  Occupational 
Analysis  Inventory  (OAI)  of  Cunningham  (1988),  both  derivatives  of  the  original  WAP,  or  the 
simplified  version  of  the  PAQ  known  as  the  Job  Element  Inventory  (JEI)  by  Cornelius  and  Hakel 
(1978).  The  conclusion  of  these  various  studies  was  that  these  job  dimensions  (factors) 
represented  the  fundamental  taxonomic  structure  of  work  (Boese  & Cunningham  1975; 
Cunningham  & Scott,  1988;  Cunningham,  Wimpee,  & Ballentine,  1990;  Harvey,  1987;  Harvey, 
Friedman,  Hakel,  & Cornelius,  1988;  Jeanneret,  1969, 1987, 1990;  Marquardt  & McCormick, 
1973;  and  McCormick,  Jeanneret,  & Mecham,  1972).  It  is  further  reasoned  that  the  stability  of 
the  factor  structure  found  with  quite  diverse  databases  is  satisfying  evidence  that  the 
methodology  (i.e.,  analyzing  jobs  with  worker-oriented  or  generalized  work  behaviors)  can 
provide  consistent  results  even  as  the  tasks  of  jobs  evolve  with  changing  technology  and  work 
redesign. 
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While  the  factor  analytic  research  using  nomothetic  questionnaires  had  significant 
influence  on  our  development  of  the  GW  As,  we  also  relied  heavily  on  job  analytic  studies  of 
managerial  and  professional  work.  In  particular,  Flanagan’s  (1951)  critical  incident  dimensions; 
William’s  (1956)  taxonomy  of  executive  performance;  the  factor  analytic  studies  of  Hemphill 
(1960),  Tomow  and  Pinto  (1976),  and  Mitchell  (1978);  and  the  integrating  studies  of  Yukl 
(1987)  and  Borman  and  Brush  (1993)  were  used  to  define  many  of  the  GWA  constructs. 
Additionally,  we  reviewed  and  drew  upon  other  specialized  job  analysis  taxonomies  of 
performance.  These  sources  included  the  Dowell  and  Wexley  (1978)  taxonomy  of  first-line 
supervisor  attributes;  Outerbridge’s  (1981)  summary  of  clusters  of  activities  related  to 
professional  level  government  employees;  O’Leary,  Rheinstein,  and  McCauley’s  (1989)  updated 
version  of  Outerbridge’s  effort;  the  Campbell,  McCloy,  Oppler,  and  Sager  (1993)  categories  from 
their  comprehensive  model  of  job  performance;  the  summary  of  general  performance 
requirements  for  all  non-managerial  jobs  in  the  U.S.  labor  force  assembled  by  Borman, 
Ackerman,  Kubisiak,  and  Quigley  (1994);  the  competency  dimension  systems  developed  by  the 
U.S.  Office  of  Personnel  Management  (1991);  activities  from  the  National  Job  Analysis  Survey 
(American  College  Testing,  1993);  and  the  summary  categories  used  for  job  activity  descriptions 
in  SCANS  (Peterson,  1994). 

Taxonomy 

The  taxonomic  structure  that  underlies  the  formulation  of  the  GWA  constructs  is  rooted 
in  the  primary  foundation  of  modem  psychology— namely  that  behavior  is  a function  of  Stimuli 
(S)  which  are  “processed”  by  Organization  (O)  which  then  lead  to  Responses  (R).  Miller  (1953) 
was  the  first  to  apply  the  S-O-R  model  to  the  study  of  work  and  subsequently  McCormick, 
Jeanneret,  and  Mecham  (1969)  used  the  paradigm  in  an  information  theory  context  to  organize 
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the  PAQ  and  the  factor  analytic  studies  of  the  job  analysis  data  obtained  with  the  PAQ.  The 
model  is  representative  of  the  three  primary  components  of  work  behavior. 

S - the  information  received  by  the  worker 

O - the  mediation  processes  performed  by  the  worker  on  the  information  received 

R - the  activities  performed  by  the  worker  in  response  to  the  “processed  stimuli” 

The  model  is  limited  when  describing  behavior  in  a work  setting,  however,  because  work 
typically  involves  interactions  and  relationships  with  others  and  occurs  within  a physical  and 
social  context.  Accordingly,  interactions  with  others  were  included  in  our  development  of  the 
GWA  constructs,  while  the  environmental  influences  were  determined  to  be  best  measured 
within  the  work  context  component  of  the  0*NET  content  model  (see  McPhail,  Blakley,  Strong, 
Collings,  Jeanneret,  & Galarza,  1995).  Thus,  our  highest-order  GWA  taxonomy  is  comprised  of 
four  broad  components  (see  Figure  6-1): 

• Information  Input 

• Mental  Processes 

• Work  Output 

• Interaction  With  Others 

A second-order  taxonomy  was  also  identified  and  it  included  nine  factors: 

• Looking  For  and  Receiving  Job-Related  Information 

• Identifying/Evaluating  Job-Relevant  Information 

• Information/Data  Processing 

• Reasoning/Decision  Making 

• Performing  Physical  and  Manual  Work  Activities 

• Performing  Complex/Technical  Activities 
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• Communicating/Interacting 

• Coordinating/Developing/Managing/Advising  Others 

• Administering 

Using  the  above  structures  as  a framework,  we  began  to  identify  the  GW  A constructs  that 
provided  coverage  of  the  domains  of  interest  from  the  previously  described  factor  analytic 
research.  Figure  6-2  presents  both  the  nine  dimension  taxonomy  and  the  42  dimension  lower- 
order  system. 

Information  Input  - Within  this  domain  we  identified  two  second-order  factors:  Looking 
for  and  Receiving  Job-Related  Information  and  Identifying/Evaluating  Job  Relevant  Information. 
Research  that  used  data  obtained  from  the  PAQ,  JEI,  and  GWI  was  the  dominant  source  for 
defining  the  GW  A constructs  that  are  necessary  to  describe  the  scope  of  the  Information  Input 
domain.  Getting  Information  Needed  to  do  the  Job  (Descriptor  #1)  and  Monitoring  Processes, 
Materials  and  Surroundings  (Descriptor  #2),  were  the  two  GWAs  that  describe  activities  of  a 
worker  when  looking  for  and  receiving  job-related  information.  Once  information  is  received,  it 
must  then  be  identified  or  evaluated.  Three  GWAs  consistently  emerged  in  the  research  literature 
that  describe  identifying/evaluating  job-relevant  information:  Identifying  Objects,  Actions,  and 
Events  (Descriptor  #3);  Inspecting  Equipment,  Structures,  or  Materials  (Descriptor  #4);  and 
Estimating  the  Characteristics  of  Materials,  Products,  Events,  or  Information  (Descriptor  #5). 

Mental  Processes  - Once  job  relevant  input  has  been  received,  the  worker’s  mental 
capabilities  are  involved  and  can  be  categorized  as  two  second-order  factors:  Processing 
Information  or  Data,  and  Reasoning/Making  Decisions.  Four  GWAs  were  identified  from  the 
research  literature  that  were  descriptive  of  Information/Data  Processing  activities:  Judging  the 
Qualities  of  Objects,  Services,  or  Persons  {Descriptor  #6);  Processing  Information  (Descriptor 
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#7);  Evaluating  Information  for  Compliance  Standards  (Descriptor  #8);  and  Analyzing  Data  or 
Information  (Descriptor  #9).  These  GW  As  emerged  from  the  analysis  of  all  levels  of  work.  On 
the  other  hand,  when  examining  the  Reasoning/Making  Decisions  second-order  factor,  the 
research  literature  indicated  that  many  of  the  relevant  constructs  were  derived  from  studies  of 
supervisory,  managerial,  or  professional  work.  More  specifically,  two  GW  As-Making  Decisions 
and  Solving  Problems  (Descriptor  #10);  and  Updating  and  Using  Job-Relevant  Knowledge 
(Descriptor  #12)— emerged  from  studies  of  all  types  and  levels  of  jobs.  Alternatively,  three  other 
GWAs  are  more  specific  to  managerial/professional  work.  These  GW  As  include:  Thinking 
Creatively  (Descriptor  #11);  Developing  Objectives  and  Strategies  (Descriptor  #13);  and 
Scheduling  Work  Activities  (Descriptor  #14).  Because  of  the  distinction  that  is  maHp  in  the 
literature  between  organizing  one’s  own  work  versus  the  activities  of  others.  Descriptor- 15 
Organizing,  Planning,  and  Prioritizing  Work  is  intended  to  include  personal  time  management, 
and  therefore,  is  likely  to  be  relevant  at  some  level  to  virtually  all  jobs. 

Work  Output  - Given  that  job-relevant  information  has  been  received  and  processed,  and 
decisions  have  been  made,  the  worker  responds  with  some  type  of  output.  Two  second-order 
factors  were  identified  which  segment  the  work  output  into  either  physical  or  technical  activities. 
The  first  second-order  factor.  Performing  Physical  and  Manual  Work  Activities  includes  four 
GWAs:  Performing  General  Physical  Activities  (Descriptor  #16);  Handling  and  Moving  Objects 
(Descriptor  #17);  Controlling  Machines  and  Processes  (Descriptor  #18);  and  Operating  Vehicles 
and  Mechanized  Devices  or  Equipment  (Descriptor  #19).  The  other  second-order  factor  was 
labeled  Performing  Complex/Technical  Activities.  While  certain  physical  movements  are 
necessarily  involved  in  the  accomplishment  of  the  specific  GWAs  subsumed  under  this  factor, 
they  -are  characterized  more  by  the  -skill  •demands  -required  for  -successful  performance.  These  six 
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GW  As  are:  Interacting  with  Computers  (Descriptor  #20);  Drafting,  Laying  Out  and  Specifying 
Technical  Devices,  Parts,  or  Equipment  (Descriptor  #21);  Implementing  Ideas,  Programs, 
Systems,  or  Products  (Descriptor  #22);  Repairing  and  Maintaining  Mechanical  Equipment 
(Descriptor  #23);  Repairing  and  Maintaining  Electronic  Equipment  (Descriptor  #24);  and 
Documenting  and  Recording  Information  (Descriptor  #25). 

Interacting  With  Others  - Communication  is,  of  course,  a critical  activity  in 
organizational  life  and  we  found  Communicating/Interacting  to  be  a useful  second-order  factor 
that  is  descriptive  of  a critical  component  of  interpersonal  relationships.  One  important  element 
of  any  communication  is  making  sure  that  the  intended  audience  understands  the  content  of  the 
communication.  Descriptor  #26,  Interpreting  the  Meaning  of  Information  for  Others,  addresses 
this  understanding  aspect  of  communication. 

The  GWA  taxonomy  draws  a distinction  between  communicating  inside  an  organization 
and  communicating  to  customers  and  others  outside  the  organization,  including  the  public. 
Communicating  With  Supervisors,  Peers,  or  Subordinates  (Descriptor  #27)  is  concerned  with  the 
internal  communication  activity;  Communicating  With  Persons  Outside  the  Organization 
(Descriptor  #28)  relates  to  external  communication  activities.  A related  activity  is  establishing 
good  working  relations  with  others,  especially  relevant  in  a team  setting,  but  more  generally 
important  for  contributing  to  a smooth  running  organization.  The  dimension  covering  this 
activity  is  labeled  Establishing  and  Maintaining  Interpersonal  Relationships  (Descriptor  #29).  An 
activity  that  is  especially  relevant  to  the  health  care  industry  and  to  elder-care  operations  is 
Descriptor  #30,  Assisting  and  Caring  for  Others. 

Still  another  interpersonal  activity,  but  of  a very  different  type  is  Selling  or  Influencing 
Others  (Descriptor -#3 1).  This  GW  A includes  persuading -others  to  buy  products  or  otherwise 
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influencing  others  to  change  their  behavior.  Similar  in  some  ways  is  Descriptor  #32,  Resolving 
Conflicts  and  Negotiating  With  Others.  However,  this  GWA  involves  handling  complaints  and 
arbitrating  disputes.  A GW  A that  at  first  appears  related  to  the  external  communication 
dimension  is  Performing  or  Working  Directly  With  the  Public  (Descriptor  #33).  Actually,  this 
activity  is  intended  to  represent  important  dimensions  of  certain  high  population  occupations 
such  as  patrol  officers,  restaurant  servers,  and  government  employees  dealing  directly  with  the 
public.  Another  second-order  factor  underlying  interaction  with  others  was  found  to  involve 
coordinating,  developing,  managing,  or  advising  others.  There  are  six  GWAs  that  we  believe 
describe  these  types  of  interrelationships.  Descriptor  #36  is  similar  to  Descriptor  #33  in  that  it  is 
targeted  toward  a particular  large  population  set  of  occupations.  Teaching  Others  (Descriptor 
#36)  involves  both  the  development  of  training  programs  and  the  delivery  of  training  or 
instruction  to  others.  Presumably,  this  dimension  is  at  the  core  of  school  teaching  jobs,  as  well  as 
trainer  positions  in  corporate  or  other  types  of  organizations. 

GWAs  34  and  35,  as  well  as  37-39  are  supervisory  or  management  dimensions.  They  are 
intended  to  cover,  in  a comprehensive  yet  parsimonious  fashion,  work  activities  required  in  a 
supervisory/managerial  setting.  Descriptor  #34,  Coordinating  the  Work  of  Others  involves 
overseeing  the  coordination  of  a team’s  or  a larger  group’s  work  activities.  A similar  GWA,  but 
with  a more  direct  team  focus,  is  Developing  and  Building  Teams  (Descriptor  #35).  The 
increasing  emphasis  in  U.S.  organizations  on  team-based  structures  motivated  this  dimension. 

A broader  GWA,  Guiding,  Directing,  and  Motivating  Subordinates  (Descriptor  #37),  is 
concerned  with  setting  standards  for  performance  and  monitoring  employee  performance  against 
those  standards.  Descriptor  #38,  Coaching  and  Developing  Others  has  overlap  with  the  teaching 
dimension  but  is  more  focused  on  providing  developmental  opportunities  for  subordinates  as  a 
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supervisor  working  directly  with  them  on  a full-time  basis.  Providing  Advice  and  Consultation  to 
Others  (Descriptor  #39)  is  not  necessarily  a management  dimension.  It  may  involve  external 
consulting  on  technical  or  management  matters. 

The  third  second-order  factor  involving  relationships  with  others  we  labeled 
administering,  and  the  last  three  GW  As  relate  more  to  administrative  than  hands-on  management 
or  supervision  dimensions.  Accomplishing  Administrative  Activities  (Descriptor  #40)  describes 
the  paperwork,  recordkeeping,  and  similar  activities  in  management  jobs.  Staffing  Organizational 
Units  (Descriptor  #41)  refers  to  the  recruiting,  selecting,  and  hiring  functions  in  an  organization. 
Finally,  Monitoring  and  Controlling  Resources  (Descriptor  #42)  involves  overseeing  non- 
personnel related  resources  such  as  budgets,  materiel,  and  other  assets. 

Figure  6-2  describes  the  42  lower  level  generalized  work  activities  (GW As)  contained  in 
the  0*NET  content  model.  Jeanneret  and  Borman  (1995)  provided  the  theoretical  and  research- 
based  rationale  for  this  taxonomy.  The  taxonomy  has  ties  to  several  other  models  or  taxonomies 
of  work  activities.  In  fact,  a cross-walk  between  our  model’s  constructs  and  the  constructs 
contained  in  the  models  reviewed  (Jeanneret  & Borman,  1995)  shows  that  our  model  is  quite 
comprehensive  and  subsumes  the  vast  majority  of  these  models’  constructs. 

Thus,  Jeanneret  and  Borman  (1995)  provided  compelling  evidence  for  the  internal, 
substantive  validity  of  the  42  dimension  system.  However,  that  work  did  not  speak  to  the 
empirical  reliability  and  validity  of  measures  of  these  dimensions.  Accordingly,  the  purpose  of 
the  research  to  be  reported  here  was  to  address  issues  of  reliability  and,  to  an  extent,  validity  of 
the  descriptor  measures. 
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Sample  and  Measures 

Previous  chapters  have  described  the  target  sample  of  occupations  for  job  analysis  and  the 
subset  of  these  occupations  that  have  actually  been  sufficiently  analyzed  to  date.  Figure  6-3  lists 
these  occupations  and  the  N for  each.  Briefly,  the  data  analyses  to  be  reported  involve  35 
occupations  with  a minimum  of  four  incumbents  per  occupation.  The  number  of  incumbents 
varies  from  four  to  88,  with  an  harmonic  mean  of  9.68  per  occupation.  Occupational  Analysis 
Field  Center  (OAFCs)  job  analysts  also  provided  ratings  on  the  GW  As.  At  least  six  analyst 
ratings  were  generated  for  each  GWA  on  each  of  the  80  occupations  in  the  entire  sample. 

Level,  importance,  and  frequency  scales  were  developed  for  the  GWA  questionnaire.  As 
in  several  other  domains,  the  level  scales  can  be  characterized  as  measuring  complexity.  The 
level  scales  are  7-point  scales  (1-7)  with  an  additional  not  relevant  option.  Three  behavioral 
statements  were  also  developed  for  each  GWA  level  scale  to  anchor  the  high,  mid-range,  and  low 
levels.  Where  possible,  we  identified  statements  from  research  on  the  Position  Analysis 
Questionnaire  (PAQ)  or  Occupational  Analysis  Inventory  (OAI)  that  had  been  previously  scaled 
according  to  level  or  complexity.  The  remaining  anchors  were  prepared  by  the  authors 
specifically  for  this  study.  The  importance  scales  have  five  points  (1-5)  with  the  verbal  anchors: 
Not  Important,  Somewhat  Unimportant,  Important,  Very  Important,  and  Extremely  Important. 

The  analyst  rating  scales  were  identical  to  those  used  by  incumbents  except  for  those 
having  to  do  with  frequency.  For  incumbents,  the  frequency  scales  are  7-point  scales  with  verbal 
anchors.  Those  anchors  are:  Once  per  year  or  less;  More  than  once  per  year;  More  than  once  per 
month;  More  than  once  per  week;  Daily;  Several  times  per  day;  and  Hourly  or  more  often.  For 
analysts,  the  frequency  scales  are  5-point  scales  with  verbal  anchors.  Those  anchors  are:  Never; 
Almost  never;  Sometimes:  Often;  and  Always. 

For  each  descriptor,  the  respondent  to  the  survey  was  asked  to  read  the  descriptor  label 
and  definition,  and  then  to  answer  the  level  scale  question,  the  importance  scale  question  next. 
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and  then  the  frequency  question.  Figure  6-4  provides  examples  of  the  level,  importance,  and 
frequency  scales. 

Results 

Descriptive  Statistics 

Table  6-1  contains  the  GWA  overall  means  across  all  35  occupations  and  the  standard 
deviations  associated  with  those  means.  Also  appearing  in  Table  6-1  are  the  interrater  agreement 
coefficients  for  each  descriptor  along  with  the  standard  error  of  measurement. 

Focusing  first  on  the  level  scales,  there  is  considerable  variation  in  the  means  across  the 
42  descriptors.  The  repair  dimensions  and  some  of  the  managerial  dimensions  have  quite  low 
means,  reflecting  the  fact  that  most  jobs  in  the  economy  (and  in  our  sample  of  35  occupations)  do 
not  involve  any  repairing  or  managing.  The  highest  means  are  for  descriptors  such  as 
Establishing  Relationships  (Descriptor  #29)  and  Communicating,  Internal  (Descriptor  #27), 
activities  that  apply  to  most  occupations.  Even  with  the  latter  dimensions,  however,  the  means 
are  only  slightly  above  the  scale  mid-point,  with  considerable  variation  across  occupations.  This 
suggests  that  range  restriction  is  not  a problem  with  these  GWA  level  scales.  A similar  pattern  of 
means  and  standard  deviations  is  evident  for  the  importance  scales.  The  frequency  scale  means 
are  almost  uniformly  higher  than  the  level  scale  means,  where  this  comparison  is  appropriate 
(i.e.,  they  both  have  7-point  scales).  The  standard  deviations  are  in  general  lower,  indicating  a 
relative  restriction-in-range  for  the  frequency  ratings. 

Reliability 

The  k-rater  interrater  agreement  Coefficients  are  for  the  most  part  quite  impressive, 
especially  for  the  level  scales.  Most  of  the  coefficients  are  in  the  .70s  and  .80s  for  the  level 
scales,  with  the  range  from  .51  to  .92  and  a median  of  .82;  for  the  importance  scales  these 
coefficients  are  usually  slightly  lower  (Mdn  = .78).  Finally,  interrater  agreement  is  lower  still  for 
the  frequency  scales  (Mdn  = .68).  Nonetheless,  overall  reliability  for  the  GWA  descriptor  scales 
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is  acceptable.  Of  course,  if  we  can  achieve  the  target  of  30  incumbents  per  occupation,  interrater 
agreement  should  be  outstanding. 

Along  these  lines.  Table  6-2  displays  the  30-rater  reliabilities  to  be  expected,  along  with 
the  1-rater  interrater  agreement  coefficients.  For  the  level  scales,  35  of  the  42  descriptors  have 
coefficients  of  .90  or  higher  at  the  30-rater  level.  As  expected,  the  importance  scales  provide 
somewhat  lower  reliabilities,  but  33  of  the  42  dimensions  have  30-rater  interrater  agreement 
coefficients  at  .90  or  higher.  Again,  the  frequency  scale  interrater  agreement  coefficients  are 
somewhat  lower,  but  at  the  30-rater  level,  these  scales  are  also  quite  reliable,  as  well.  In 
summary.  Tables  6-1  and  6-2  demonstrate  that  the  generalized  work  activity  requirements  of 
occupations  can  be  reliably  evaluated  by  incumbents  in  those  occupations.  As  a benchmark,  the 
reader  is  referred  to  the  work  of  Geyer,  Hice,  Hawk,  Boese,  & Brannon  (1989)  who  the 
reliabilities  of  four  experienced  occupational  analysts  across  20  diverse  occupations  using  the 
standard  United  States  Employment  Services  job  analysis  procedures.  For  work  functions  (data, 
people,  things),  the  reported  reliabilities  ranged  from  .77  (one  rater)  to  .95  (four  raters)  using 
coefficient  alpha  and  a variance  ratio  procedure  to  measure  analyst  consistency. 

Scoring 

As  discussed  in  previous  chapters,  it  might  be  argued  that  including  not  relevant  (i.e., 
zero)  scores  when  computing  interrater  agreement  provides  estimates  of  reliability  higher  than  if 
these  were  not  included.  This  argument  could  be  extended  to  the  importance  and  frequency 
scales  in  that  when  not  relevant  was  indicated  on  the  level  scale,  a 1 (not  important  or  once  per 
year  or  less)  rating  was  employed  in  computing  the  interrater  agreement  for  the  importance  and 
frequency  scales.  To  address  the  possibility  of  different  reliability  estimates  for  these  different 
-scoring  procedures,  we  recalculated  the  reliabilities  with  not  relevant  responses  removed.  Finally 
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for  the  level  scales  only,  reliabilities  were  computed  using  a simple  relevant/not  relevant  coding 
scheme. 

Interrater  agreement  coefficients  calculated  using  these  alternative  coding  systems  appear 
in  Table  6-3.  For  the  level  and  frequency  scales  the  interrater  agreement  coefficients  are  not 
affected  when  the  not  relevant  option  is  not  considered.  They  are  virtually  the  same  as  when  not 
relevant  is  scored  as  zero.  For  importance,  the  changes  in  reliabilities  are  very  small;  in  several 
cases,  interrater  agreement  actually  improves  slightly  when  the  not  relevant  response  on  the  level 
scale  is  ignored.  When  the  level  scales  are  scored  dichotomously,  as  relevant  or  not  relevant, 
some  of  the  descriptors  suffer  a substantial  loss  of  reliability,  but  others  lose  very  little,  and  in 
four  cases,  reliabilities  actually  improve. 

From  both  a reliability  and  a conceptual  perspective,  the  full  scale  scoring  method 
employing  the  not  relevant  response  seems  preferable.  Interrater  agreement  is  somewhat  better 
for  the  importance  ratings  when  the  not  relevant  (zero)  scale  point  is  included.  In  addition,  the 
case  can  certainly  be  made  that  a not  relevant  rating  for  a GWA  is  conceptually  quite  different 
from  a low  level  requirement  for  that  GWA. 

Analyses  of  Variance 

An  alternative  view  of  interrater  agreement  can  be  derived  from  analysis  of  variance 
(ANOVA).  ANOVA  provides  summary,  across  descriptor  indices  of  interrater  agreement. 

Results  of  these  analyses  are  in  Tables  6-4a,  6-4b,  and  6-4c,  for  the  level,  importance,  and 
frequency  scales  respectively.  Results  are  quite  similar  for  the  three  types  of  scales.  First,  the 
significant  occupations  effect  indicates  that  occupations  are  in  fact  differentiated  by  these 
descriptors.  The  significant  descriptor  effect  confirms  what  we  reported  in  Table  6-1.  Means  for 
the  descriptors  across  the  35  occupations  vary  considerably.  Finally,  the  significant  descriptor  by 
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occupations  interaction  provides  evidence  that  different  patterns  of  GWA  requirements  are 
evident  for  different  occupations. 

The  intraclass  interrater  agreement  coefficients  summarizing  the  ANOVA  results  appear 
in  Table  6-5.  The  k-rater  coefficients  for  the  three  types  of  scales  are  .80,  .78,  and  .74.  The  30- 
rater  interrater  agreement  coefficients  are  .90  or  above.  These  findings  indicate,  again,  that 
interrater  agreement  for  the  GW  As  is  quite  high,  and  very  consistent  with  occupational  analyst 
reliability  found  by  other  researchers  (Geyer,  et  al.,  1989;  McCormick  & Jeanneret,  1988). 

Tables  6-6a,  6-6b,  and  6-6c  present  ANOVA  results  at  the  aggregate  construct  level-that 
is,  for  the  mean  scores  computed  across  the  GWA  descriptors  found  within  the  nine-dimension 
higher-order  taxonomy.  The  findings  for  the  occupations,  descriptor,  and  descriptor  x 
occupations  effect  are  highly  similar  to  what  was  found  for  the  lower  level,  42-construct  system. 
All  three  of  the  effects  are  highly  significant  for  each  of  the  scale  types.  Further,  Table  6-7  shows 
the  intraclass  interrater  agreement  coefficients  at  this  aggregated  level.  The  k-rater  coefficients 
are  higher  by  4-6  correlation  points  than  the  coefficients  computed  for  the  lower  level  42- 
dimension  system.  The  1 -rater  and  30-rater  coefficients  are  similarly  larger  for  the  aggregated  9- 
dimension  system  than  for  to  the  42-descriptor  system.  Accordingly,  the  GWA  descriptors  appear 
to  have  very  good  reliability,  either  at  the  individual  scale  or  the  aggregated  scale  level. 

Descriptor  and  Scale  Relationships 

As  with  other  domains  in  the  content  model,  we  investigated  redundancies  among  the 
level,  importance,  and  frequency  scales.  Although  there  are  clear  conceptual  distinctions  between 
the  scales,  empirical  redundancies  are  certainly  possible.  Table  6-8  summarizes  correlations 
between  the  three  types  of  scales  computed  in  two  different  ways.  First,  a correlation 
between  each  pair  of  scale  types  was  computed  for  each  dimension  (at  the  42  dimension  level) 
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across  the  35  occupations  and  then  averaged  over  the  42  dimensions.  A second  mean  correlation 
was  derived  by  computing  a between-scale  correlation  for  each  occupation  across  the  42 
dimensions  and  averaging  over  the  35  occupations.  Correlations  are  high  between  the  level  and 
importance  scales,  .92  and  .93  for  the  two  approaches  to  data  averaging.  The  standard  deviations 
for  these  mean  correlations  are  quite  a bit  higher  when  the  correlations  are  computed  across  the 
42  descriptors  within  occupation  and  averaged  across  the  35  occupations.  Apparently,  for  some 
occupations,  the  level-importance  relationship  is  lower.  The  corresponding  importance-frequency 
mean  correlations  are  somewhat  lower  than  the  level-importance  ratings  (.89  and  .91),  but  still 
very  high.  The  level-frequency  mean  correlations  are  definitely  lower  (.82  and  .88).  Overall, 
Table  6-8  suggests  considerable  redundancy  for  the  level  and  importance  scales.  The  frequency 
scale  demonstrated  less  redundancy,  particularly  with  the  level  scale. 

Tables  6-9a,  6-9b,  and  6-9c  present  correlations  between  dimensions  for  the  level, 
importance,  and  frequency  scales.  These  correlations  are  at  the  occupation  level,  with  the  N for 
each  correlation  equal  to  35.  Tables  6-lOa,  6-10b,  and  6-10c  contain  between-dimension 
correlations  at  the  individual  level.  Because  the  focus  of  the  present  research  is  to  study 
occupations  and  differences  between  them,  we  will  focus  on  the  Table  6-9  relationships.  In 
addition,  as  the  patterns  of  relationships  are  similar  for  level,  importance,  and  frequency,  so  our 
attention  will  be  primarily  on  the  level  results. 

Correlations  in  Table  6-9a  for  the  most  part  make  good  intuitive  sense.  Perhaps  most 
impressive,  as  with  several  other  domains,  the  mean  correlations  between-descriptors,  within 
higher-order  constructs  (e.g..  Looking  For  and  Retrieving  Job-Related  Information  in  Figure  6-2) 
is  .61,  whereas  the  mean  between-descriptor,  across  higher-order  construct  correlation  is  .43.  As 
.an  example,  for  the  Administering  higher  level  composite,  foe  -three  GWAs  in  this  composite 
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(Performing  Administrative  Tasks  [Descriptor  #40],  Staffing  Organizational  Units  [Descriptor 
#41]  and  Monitoring  Resources  [Descriptor  #42])  intercorrelate  .55,  .55,  and  .77.  The  negative 
correlations  between  descriptors  are  also  intuitively  appealing.  For  example.  Controlling 
Machines  (Descriptor  #18)  and  Performing  Administrative  Tasks  (Descriptor  #40)  correlate  -.50; 
Performing  Physical  Work  Tasks  (Descriptor  16)  and  Processing  Information  (Descriptor  #8) 
correlate  -.60.  The  sense  of  these  negative  relationships  is  that  for  several  of  these  pairs  of 
dimensions,  occupations  require  either  one  of  the  dimensions  or  the  other,  but  seldom  both. 
Overall,  the  GWA  scales  provide  a coherent  and  meaningful  pattern  of  correlations. 

Factor  Structure 

Examining  the  patterns  of  correlations  in  Table  6-9  is  useful  for  evaluating  the  rationality 
of  these  relationships.  However,  a more  comprehensive  and  efficient  approach  is  to  conduct  an 
exploratory  factor  analysis  of  the  correlation  matrix.  Accordingly,  we  accomplished  this  with  a 
principal  components  analysis  and  an  orthogonal  rotation  of  the  correlations  between  the  level 
scales  in  Table  6-9a.  In  particular,  the  two  through  five  rotated  factor  solutions  were  examined, 
and  the  three-factor  solution  proved  to  be  the  most  interpretable.  This  solution  is  presented  in 
Table  6-11.  The  three  factors  accounted  for  71%  of  the  total  variance  for  ratings  on  the  level 
scale.  Also,  the  communalities  suggested  that  for  the  most  part  the  GWA  dimensions  are  well- 
represented  in  this  solution.  The  first  factor  we  called  working  with  information.  It  accounts  for 
48%  of  the  total  variance. 

The  second  factor  has  many  of  the  supervisory  and  working  with  others  GW As  loading 
on  it;  the  factor  is  labeled  working  with  and  directing  the  activities  of  others  and  accounts  for 
14%  of  the  variance.  The  third  factor  was  named  manual  and  physical  activities:  performing 
repair  and  other  physical  work,  and  it  accounts  for  8%  of  the  variance. 
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The  three-factor  solution  lines  up  fairly  well  with  the  highest  order  4-dimension  system  in 
the  Jeanneret  and  Borman  (1995)  hierarchical  taxonomy  (see  Figure  6-1).  The  working  with 
others  factor  overlaps  considerably  with  the  interacting  with  others  dimension,  and  the  manual 
and  physical  activities  factor  contains  many  of  the  work  output  GW  As.  Factor  1,  working  with 
information,  overlaps  with  much  of  the  information  input  and  mental  processes  part  of  the  GW  A 
model.  Overall,  this  3-factor  system  summarizes  well  the  GWA  domain.  The  factors  are  highly 
inteipretable  and  almost  all  of  the  GW As  are  represented  in  the  solution. 

As  mentioned  in  previous  chapters,  we  must  remember  that  the  factor  analysis  is  based  on 
a small  N (N  = 34  in  this  domain).  This  may  create  less  than  a stable  solution.  However,  the 
inteipretability  of  these  results  argues  against  this  conclusion. 

A more  direct  test  of  the  viability  of  the  3-dimension  highest  order  or  the  9-dimension 
systems  is  to  conduct  confirmatory  factor  analyses.  We  attempted  these  analyses,  but  the 
solutions  did  not  converge.  This  is  not  surprising,  given  the  small  number  of  occupations  in 
comparison  to  the  number  of  variables. 

Occupation  Differences 

An  important  application  of  the  0*NET  data  will  be  to  provide  profiles  of  occupational 
requirements  on  the  0*NET  descriptors.  Accordingly,  we  present  the  GWA  profiles  for  six 
different  occupations  selected  to  be  representative  of  very  different  types  of  employment  This 
should  provide  an  initial  view  as  to  what  the  profiles  might  look  like  in  the  GWA  domain.  Tables 
6-12a,  6-12b,  and  6-12c  display  the  means  and  standard  deviations  for,  respectively,  the  level, 
importance,  and  frequency  scales  relative  to  these  six  occupations.  A summary  observation  is 
that  the  GW  As  appear  to  describe  these  occupations  quite  accurately.  The  similarities  and, 
.especially,  the  differences  between  occupationsseem  appropriate, -given  the  mature  of  these 
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occupations.  For  example,  for  the  Performing  Physical  Work  Tasks  (Descriptor  16)  GW  A,  means 
for  the  General  Managers  and  Computer  Programmers  are  very  low;  substantially  higher  means 
are  evident  for  Patrol  Officers,  Janitors,  and  Repairers.  For  Operating  Vehicles  (Descriptor  #20), 
as  would  be  expected,  the  highest  mean  value  is  for  Patrol  Officers,  whereas  the  Computer 
Programmer  mean  is  literally  zero  (not  relevant).  Also,  as  expected.  General  Managers  have  the 
highest  mean  on  Directing  Subordinates  (Descriptor  #37),  and  Managers  and  Patrol  Officers  are 
highest  on  the  Resolving  Conflict  (Descriptor  #32)  GW  A.  The  profiles,  for  both  occupations 
(across  GW As)  and  GW  As  (across  these  occupations)  seem  to  offer  an  appropriate  and  useful 
picture  of  the  GWA  requirements  for  occupations. 

Some  of  the  standard  deviations  are  instructive  as  well.  For  example,  one  of  the  highest 
standard  deviations  is  for  Providing  Consultation  (Descriptor  #39)  relative  to  Computer 
Programmers.  It  is  quite  possible  this  reflects  actual  differences  in  the  respondents’  jobs,  such 
that  some  Programmers  work  alone  and  others  consult.  Thus,  in  some  cases,  the  standard 
deviations  may  be  interpretable  and  provide  useful  occupational  information. 

A quite  different  way  to  explore  occupation  differences  in  GW As  is  to  compare  patterns 
of  GWA  ratings  within  occupation  to  the  patterns  across  occupations.  A very  simple  way  to 
address  occupation  differences  in  this  regard,  at  least  initially,  is  to  identify  a small  number  of 
occupations  for  which  we  have  a relatively  large  incumbent  sample  size,  split  the  samples  in  two 
within  each  of  the  occupations,  and  then  correlate  the  mean  incumbent  ratings  within  and  across 
occupations.  This  was  accomplished  for  three  occupations:  First-Line  Supervisors;  Secretaries 
(Except  Legal  and  Medical);  and  General  Office  Clerks. 

The  six  by  six  correlation  matrix  (3  occupations  x 2 samples  for  each)  appears  in  Table  6- 
17.  As  can  be  seen,  the  within-occupation,  split-half  correlations. are  .96,  .96.,  and  .98,  and  these 
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are  all  higher  than  any  of  the  across-occupation  correlations.  The  Secretaiy-Clerk  correlations  are 
almost  as  high  as  the  correlations  within  each  of  these  occupations,  however.  This  is  not 
surprising,  as  these  occupations  would  be  expected  to  require  very  similar  work. 

Discriminant  Analysis 

A central  objective  for  the  GWA  descriptors  is  to  discriminate  between  occupations.  A 
preliminary  look  at  how  well  they  accomplish  this  was  provided  by  the  results  in  Table  6-13  and 
our  discussion  of  the  six  occupations’  standing  on  various  GW  As.  Also,  the  Table  6-17  results 
showed  that  the  patterns  of  mean  ratings  within  occupation  are  more  similar  than  these  patterns 
across  occupations.  A more  systematic  way  to  assess  the  GWA’s  ability  to  differentiate 
occupations  is  to  conduct  a discriminant  function  analysis  on  all  35  occupations.  The 
discriminant  analysis  also  indicates  the  GW  As  that  contribute  most  (and  least)  to  this 
differentiation.  The  1-5  function  solutions  were  examined  and  the  4-function  solution  was 
selected  as  most  interpretable.  Table  6-13  presents  the  rotated  correlations  between  the 
discriminating  variables  and  the  four  canonical  discriminant  functions.  These  vectors  of 
correlations  should  be  interpreted  as  the  loadings  on  the  GWA  level  scales  that  maximally 
differentiate  between  the  35  occupations.  The  data  in  Table  6-13  also  show  for  each  GWA,  a2 
coefficients  that  summarize  how  much  discriminating  variance  each  GWA  level  scale  is 
providing. 

The  first  function  is  clearly  a supervisory  or  management  variable.  Apparently,  a clear-cut 
and  important  distinction  between  occupations  in  our  sample  is  whether  or  not  they  are 
management  or  non-management  jobs.  The  second  function  involves  working  with  data;  the  third 
function  is  primarily  concerned  with  operating  vehicles;  and  the  fourth  function  has  some 
■similarity  to  die  manual  and  physical  activities  factor  inthe  factor  analysis,  with  repair  and 
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inspecting  activities  loading  on  it.  Thus,  for  the  most  part,  the  discriminant  analysis  yielded 
interpretable  functions  describing  the  GWA-related  features  of  occupations  that  provide 
maximum  differentiation  between  the  occupations. 

The  discriminant  analysis  functions  were  able  to  correctly  classify  62%  of  the  incumbents 
into  their  occupations,  when  all  35  functions  are  used,  and  38%  when  only  the  four  retained 
functions  are  used.  Both  of  these  classification  rates  are  well  above  chance,  which  would  be  3% 
correct  classification  for  35  occupations,  if  the  occupations  were  of  equal  size.  The  T(2 
importantly  show  that  with  a few  exceptions  the  GW  As  each  provide  a fair  amount  of 
differentiation.  These  T|2  vary  from  .10  to  .40.  The  GW  As  that  have  the  smallest  r\2  are  Handling 
Objects  (Descriptor  #17)  (.10),  Estimating  Characteristics  (Descriptor  #5)  (.13),  Interpreting 
Information  (Descriptor  #26)  (.  14),  and  Documenting  Information  (Descriptor  #25)  (.14). 
Interestingly,  these  are  the  four  least  reliable  level  scales.  These  findings,  along  with  the 
occupation  differences  data  in  Table  6- 12a,  suggest  that  GW  As  will  be  an  important  domain  for 
providing  information  about  job  requirements  that  differentiate  between  occupations. 

Convergence  with  Analysts’  Ratings 

As  mentioned  previously,  OAFC  occupational  analysts  provided  ratings  on  GW  As  and 
several  other  content  model  domains  for  all  80  occupations  in  the  target  sample.  In  this  section, 
we  compare  their  GW  A ratings  with  the  incumbent  ratings  on  the  35  occupations  evaluated  by 
both  the  incumbents  and  the  analysts.  Tables  6- 14a  and  6-  14b  present  these  comparisons  for  the  . 
level  and  importance  scales,  respectively.  These  comparisons  were  not  appropriate  for  frequency 
scales  because  incumbents  and  analysts  used  different  frequency  scales.  Tables  6- 14a  and  6- 14b 
provide  means,  standard  deviations,  and  k-level  reliabilities  for  both  data  sources.  Also  provided 
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are  t and  F tests  comparing  the  means  and  standard  deviations  of  incumbent  and  analyst  data,  as 
well  as  the  correlation  between  the  two  sets  of  ratings  for  each  GW  A,  and  a d statistic  indicating 
the  average  squared  difference  between  analyst  and  incumbent  mean  ratings. 

Regarding  results,  the  interrater  agreement  within-source  findings  indicate  that  analyst 
ratings  are  about  .10  points  higher  than  those  for  the  incumbents  (Mdn  t = .93  and  .82).  The 
analyst  GW  A level  ratings  are  highly  reliable,  with  no  need  to  have  additional  analysts  provide 
these  ratings.  Reliability  results  for  the  PAQ  job  dimensions  provide  benchmark  data  for  analyst 
ratings  that  are  relevant  to  many  of  our  GW  As.  McCormick,  Mecham,  & Jeanneret  (1989)  report 
the  reliability  of  job  dimension  scores  calculated  from  43  studies  that  involved  19,961  analyst 
pairs.  The  median  of  the  median  reliability  coefficients  across  all  45  PAQ  dimensions  was  .91.  If 
we  examine  just  the  reliabilities  for  those  PAQ  dimensions  that  are  “matched”  to  the  GW  As, 
there  are  20  such  dimensions  with  median  reliabilities  that  range  from  .84  to  .97  and  an  overall 
median  of  .915.  Correlations  between  the  two  sources  (i.e.,  incumbents  and  analysts)  are 
reasonably  high.  The  median  correlation  between  the  mean  level  ratings  is  .70.  Looking  at 
agreement  in  a correlational  sense,  the  lowest  agreement  between  analysts  and  incumbents  is  on 
the  Handling  Objects  (Descriptor  #17)  and  Establishing  Relationships  (Descriptor  #29)  GW  As. 
The  latter  case  is  curious  because  the  reliabilities  within  source  are  reasonably  high;  incumbents 
and  analysts  are  in  good  agreement  among  themselves  on  how  to  order  occupations  on  this 
dimension,  but  these  views  are  fairly  idiosyncratic  to  their  own  source.  More  broadly,  however, 
the  correlational  results  for  the  level  scales  suggest  substantial  agreement  across  the  two  sources 
in  the  rated  patterns  of  occupational  GW  A requirements  for  this  sample  of  occupations. 
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Another  view  of  the  similarities  and  differences  between  the  two  sets  of  ratings  can  be 
obtained  by  comparing  the  factor  structures  of  the  incumbents’  and  analysts’  data.  Table  6-16 
depicts  the  3-factor  varimax  rotated  solution  for  the  analyst  ratings.  This  solution  is  very  similar 
to  the  3-factor  solution  emerging  from  the  incumbent  data  (see  Table  6-11).  The  labels  for  the 
incumbent  factors  are  very  appropriate  for  these  factors  as  well.  Thus,  from  the  standpoint  of 
relationships  between  the  descriptors,  i.e.,  what  dimensions  are  perceived  as  similar  to  and 
different  from  each  other,  there  is  also  good  agreement  between  incumbents  and  analysts. 

Although  correlational  agreement  and  structural  similarity  are  high,  many  significant 
level  differences  were  found  between  incumbents’  and  analysts’  ratings.  The  t-tests  indicate  that 
a little  more  than  half  of  the  GWAs  (23)  show  significant  differences  between  the  two  sources’ 
ratings.  In  seven  cases,  the  analysts’  means  are  significantly  higher  than  the  incumbents’  means. 
However,  for  16  of  the  GWAs  the  incumbent  means  are  higher,  with  some  substantial 
differences.  For  example,  the  Using  Job  Knowledge  (Descriptor  #12)  means  for  the  incumbent 
and  analyst  groups  are  3.80  and  2.96.  For  Organizing  and  Planning  (Descriptor  #15),  the 
corresponding  means  are  3.80  and  2.73.  Thus,  on  several  GWAs,  the  incumbents  believe  that  the 
GWA  requirements  are  at  a higher  level  than  do  the  analysts.  Essentially,  incumbents  are 
reporting  that  their  job  is  more  complex  than  is  seen  by  the  analysts. 

For  the  importance  ratings  (Table  6-14b),  the  across-source  correlations  are  not  quite  as 
high  as  for  the  level  ratings.  However,  the  real  differences  between  incumbents  and  analysts  is 
seen  by  looking  at  the  means.  The  level  trend  is  substantially  reversed,  with  many  of  the  GWAs 
having  significantly  higher  means  for  the  analysts.  Analysts  see  dimensions  of  these  occupations 
as  less  complex  but  more  important  than  do  the  incumbents.  Overall,  the  correlational  results 
show  reasonable  convergent  validity  for  the  incumbent  and  analyst  ratings,  especially  for  the 
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level  scales.  However,  significant,  and  in  some  cases  substantial,  differences  in  means  are 
evident  between  the  two  sources. 

Another  way  to  examine  incumbent-analyst  differences  in  the  ratings  is  to  treat  the  level 
scale  as  dichotomous  (i.e.,  is  the  dimension  seen  as  relevant  or  not  relevant  for  an  occupation?), 
and  evaluate  the  differences  between  means  on  this  zero  (not  relevant)  to  one  (relevant)  scale. 
Table  6-15  contains  means,  standard  deviations,  within  source  reliabilities,  and  t statistics 
indexing  the  significance  of  the  differences  between  means.  Surprisingly,  the  means  for  the 
analysts  are  almost  all  greater  than  those  for  the  incumbents.  Recall  that  the  incumbent  means 
tend  to  be  higher  than  the  analyst  means  when  the  entire  level  scale  is  considered.  Apparently, 
analysts  are  considerably  less  likely  than  incumbents  to  use  the  not  relevant  option.  For  example, 
the  means  are  .98  or  higher  for  GW  As  1,  2,  3,  7,  8,  9,  10,  12,  15,  17,  27,  and  29.  The  highest 
mean  for  the  incumbents  is  .90  (for  Communicating,  Internal  [Descriptor  27]). 

Because  of  this  relative  restriction-in-range  for  analyst  ratings  on  several  of  the 
dimensions,  interrater  agreement  between  analysts  was  lower  on  average  than  the  agreement 
between  incumbents.  The  median  k-rater  reliability  was  .70  for  incumbents  and  .62  for  the 
analysts.  Thus,  if  a dichotomous  relevant-not  relevant  scale  were  to  be  employed  in  0*NET,  for 
GW  As  the  incumbents’  data  would  be  considerably  more  useful  than  analyst  data  from  the 
standpoint  of  both  reliability  of  responses  and  the  balanced  use  of  the  relevant  and  not  relevant 
response  options. 

Conclusions 

Research  reported  in  this  chapter  indicated  that  job  incumbents  using  the  GWA 
descriptors  could  reliably  describe  their  jobs.  Interrater  agreement  was  quite  high,  even  for  the  k- 
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level  reliabilities,  where  k fell  considerably  short  of  the  target  30  incumbents  per  occupation.  For 
GW  As,  at  least,  this  target  might  be  reduced  somewhat  based  on  these  results.  Of  the  three  types 
of  scales,  the  behaviorally  anchored  level  scales  showed  the  highest  reliabilities,  slightly  higher 
than  for  the  importance  scales,  and  substantially  higher  than  those  obtained  with  the  frequency 
scales. 

Relationships  between  the  level  and  importance  scales  were  high  (above  .90).  The  level 
scales  did  not  correlate  as  highly  with  the  frequency  scales.  The  high  level-importance  scale 
correlations  argue  for  possibly  dropping  one  of  these  scales  for  future  research  and 
implementation.  If  a scale  is  to  be  dropped,  we  recommend  eliminating  the  importance  scale.  The 
level  scales  are  somewhat  more  reliable  and  the  behavioral  statements  anchoring  the  level  scales 
appear  to  clarify  the  definitions  of  these  scales. 

The  present  research  tends  to  support  the  Jeanneret  and  Borman  (1995)  taxonomy. 
Regarding  the  hierarchical  structure  proposed  in  that  taxonomy,  data  from  the  level  scale  ratings 
indicate  that  the  mean  correlation  between-descriptor,  within  higher-order  construct  (e.g., 
Looking  for  and  Retrieving  Job-Related  Information)  is  substantially  higher  than  the  mean 
between-descriptor.  across  higher-order  construct  correlation  (r=  .61  versus  .43).  This  provides 
support  for  the  particular  hierarchical  system  proposed  for  the  GW As.  Also,  a factor  analysis  of 
the  between-descriptor  correlations  yielded  a readily  interpretable  3-factor  solution,  further 
evidence  for  the  coherence  of  this  GWA  taxonomy. 

Especially  important  for  evaluating  the  usefulness  of  0*NET  with  respect  to  the  GW  As, 
a discriminant  function  analysis  showed  that  the  vast  majority  of  GWA  dimensions  contributed 
substantially  to  the  differentiation  between  the  35  occupations  in  this  research.  Accordingly,  the 
GWA  descriptors  accomplished  what  was  intended  by  describing  activity  requirements  that 
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differentiate  occupations.  Further,  the  GWA  profiles  for  the  six  example  occupations 
demonstrated  in  a concrete  way  how  useful  these  profiles  can  be  for  documenting  differences 
between  occupations. 

As  argued  in  previous  chapters,  we  are  reluctant  to  drop  descriptors  from  the  taxonomy. 
The  42-dimension  system  has  a strong  theoretical  and  conceptual  rationale.  The  dimensions  were 
carefully  developed  to  provide  a comprehensive  yet  parsimonious  depiction  of  GWA 
requirements  in  Occupations.  Nonetheless,  the  Handling  Objects  (Descriptor  #17),  Estimating 
Characteristics  (Descriptor  #5),  and  Documenting  Information  (Descriptor  #25)  GW  As  might  be 
candidates  for  either  dropping  or  revising  based  on  our  results  to  date.  These  dimensions  were 
the  least  reliable  and  also  played  a minimal  role  in  differentiating  between  occupations.  It  should 
be  recognized,  however,  that  these  results  are  based  on  only  35  occupations  at  this  point. 

In  sum,  the  proposed  GWA  taxonomy  received  considerable  support  in  this  research.  The 
scales  measuring  dimensions  of  the  taxonomy  provided  reliable,  coherent,  and  useful 
occupational  information.  The  GWA  taxonomy  and  measures  of  its  dimensions  appear  to  provide 
a viable  system  for  describing  similarities  and  differences  between  occupations. 
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Highest  Order  GW  A Taxonomy 
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Figure  6-2  (continued) 

Descriptions  and  Definitions  of  Generalized  Work  Activities 
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Figure  6-2  (continued) 

Descriptions  and  Definitions  of  Generalized  Work  Activities 
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Figure  6-2  (continued) 

Descriptions  and  Definitions  of  Generalized  Work  Activities 
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Figure  6-2  (continued) 

Descriptions  and  Definitions  of  Generalized  Work  Activities 
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Figure  6-2  (continued) 

Descriptions  and  Definitions  of  Generalized  Work  Activities 
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Figure  6-2  (continued) 

Descriptions  and  Definitions  of  Generalized  Work  Activities 
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Figure  6-2  (continued) 

Descriptions  and  Definitions  of  Generalized  Work  Activities 
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Figure  6-2  (continued) 

Descriptions. and  Definitions  of  Generalized  Work  Activities 
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Figure  6-2  (continued) 

Descriptions  and  Definitions  of  Generalized  Work  Activities 
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Figure  6-2  (continued) 

Descriptions  and  Definitions  of  Generalized  Work  Activities 
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Figure  6-2  (continued) 

Descriptions  and  Definitions  of  Generalized  Work  Activities 
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Figure  6-2  (continued) 

Description  1 Definitions  of  Generalized  Work  Activities 


cd 

o 

CO 

13 

> 


O <D 

J2  J2 

u u 

B-.S 

<u  -o 
c >% 


<D 


j=  -r; 


O .£P 


§>  ■* 


o 

JC 

<d 


o > 

^ CO 

to  a. 

.£  S 

Q_.  co 

f g 

Q_>  * — 
Sr  4—* 

<u  — 

a a 


j= 

60 


O § 
5/3  "G 

13 

3 gs 
S 3 
> ,8 
•o  53 

JS  53 


to 

c 


O 

£ & 
ii 

^ j= 

fr-g 

«D  5 
^ w 
W)  CL 

.£  S 

CL  6/3 

lg 

<D  •— 
£ « 
q a 


O 


3 

o 

o !g 

<D  . 

c o 

cd  *3 

■4— » ,3 

C/3  <4-4 

S £ 

CO  QJ 

cd  £3 

° J. 

£2 
cd  00 
o .Cr 


:=  O 

S 2 ~ 

> d cd 
O .£  3 
*3  ; G 
Oh  O co 


O 

3 
13 
j s 

13 

E 


<D  co 

s fe 

> -c 

2 o 

CL  o 

2 O 

to  y 


"O  to 

<L>  .52 

<D  CO 

Z CO 

cd 

1 


J= 

to 


£ 

o 

nJ 


to  5 

I | 

« . s 

w S -S 

t>  .33  r*"* 

CL 


O 

13  .« 
> 3 
2 o 

•4  o 
s £ 

o J 

-1 

cd  3 

60  8 
c o 
o « 
Q 2 


■§> 

£ 


u 

> 

cd 

to  ai 

=5  S 
« :§ 
3 2 
co  3 

5— i rH 

<L> 

CL 

g)  8 

[>  C 

1 S 

c o 


co 

S> 

o .> 

bO  u 

.5  3 

53 

cd  o 

3 m 

a § 

CL  £ 


r O 


o 


Zh 

"O  t> 
<u  »- 
oj  cd 

3 >v 

-2  £ 

33  .E 

co  co 

*0  <3 
c j= 
<d  g 


*—  CL 

E 

.S  o 
o <-> 
Q 8 


d> 

o 

c 

> 

c 

o 

o 


3 


<D 

Q 

cd 

c 

O 

cd 

1— 

<D 

CL 

O 


<D 

> 


0> 

o e 

O <D 
O JC 

o 

gf; 

a ‘5 


cl 

1E 


c 

o 


o 

S>| 

.5  2 

Dm  tO 
2 3 
13  3 

> t3 

o o 
Q ^ 


o 

a 

cd 

o 


o 

co 

i-4 

<L> 

CL 


<D 

O 

C 

cd 

4-^ 

co 

’00 

CO 

cd 

to 


•g  a 

2 «£ 

Ol  O 


to 

c 

*5) 

c 

cd 

j= 

o 


<D 

JD  ^ 

O o g 

w co  O 

co  -a  .3 
u*  jr  4-4 

5 g o 

^ bp  u 
0 ^3  o 
bO  co  co 

.5  ‘-5  *2 

o c 3 
c cd  2 

111 

o V <u 

U E£ 


4> 

X5 

cd 

o 

2 


o 

U 


to 

c . 

■3  CO 

.£  CL 

fS  s 

I s 

cd  o 

E i 

•o  — 

g £ 

to  cd 
c c 

E g 

co  1— 

*0  B- 

rrt  L-i 


W .£ 


ON 

cs 


CD 

-C 

o 

Ui 

£ 

to 

G 

*c 

cd 

o 

■o 

G 

cd 

to 


co 

co 

< 

o 

CO 


<D 

-G 

O 

to 


flj 

3 

c 


o 

to 


flj 

CO 


CO 


L^3 


•CO 

in 


Figure  6-2  (continued) 

Descriptions  and  Definitions  of  Generalized  Work  Activities 
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Figure  6-2  (continued) 

Descriptions  and  Definitions  of  Generalized  Work  Activities 
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Figure  6-2  (continued) 

Descriptions  and  Definitions  of  Generalized  Work  Activities 
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Figure  6-2  (continued) 

Descriptions  and  Definitions  of  Generalized  Work  Activities 
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Figure  6-3 

Thirty-five  Occupations  With  Four  or  More  Incumbents  Completing  the  Generalized  Work  Activities 
Questionnaire 


Occupation 

Code 

Occupation  Title 

Number  of 
Respondents 

15005 

Education  Administrators 

11 

19005 

General  Managers  & Top  Executives 

43 

21108 

Loan  Officers  & Counselors 

6 

22135 

Mechanical  Engineers 

11 

25105 

Computer  Programmers 

7 

31302 

Teachers,  Preschool 

6 

31305 

Teachers,  Elementary  School 

13 

32502 

Registered  Nurses 

25 

32902 

Medical  & Clinical  Laboratory  Technologists 

8 

49008 

Salespersons,  Except  Scientific  & Retail 

13 

49011 

Salespersons,  Retail 

18 

49021 

Stock  Clerks,  Sales  Floor 

13 

49023 

Cashiers 

22 

51002 

1st  Line  Supervisors,  Clerical  & Administrative 

59 

53102 

Tellers 

4 

53311 

Insurance  Claims  Clerks 

7 

53905 

Teachers’  Aides  & Assistants,  Clerical 

8 

55108 

Secretaries,  Except  Legal  & Medical 

65 

55305 

Receptionists  & Information  Clerks 

5 

55338 

Bookkeeping,  Accounting  & Auditing  Clerks 

25 

55347 

General  Office  Clerks 

88 

! 61005 

Police  & Detective  Supervisors 

13 

63014 

Police  Patrol  Officers 

24 

65008 

Waiters  & Waitresses 

10 

65026 

Cooks,  Restaurant 

4 

65038 

Food  Preparation  Workers 

27 

66008 

Nursing  Aides,  Orderlies  & Attendants 

22 

67005 

Janitors  & Cleaners 

30 

85119 

Other  Machinery  Maintenance  Mechanics 

4 

85132 

Maintenance  Repairers,  General  Utility 

27 

87902 

Earth  Drillers,  Except  Oil  & Gas 

4 

89108 

Machinists 

4 i 

92974 

Packaging  & Filling  Machine  Operators 

16 

97102 

Truck  Drivers,  Heavy  or  Tractor  Trailer 

9 

97111 

Bus  Drivers,  Schools 

11 

Figure  6-4 

Example  Page  From  the  Generalized  Work  Activities  Questionnaire 


23.  Repairing  and 
Maintaining 
Mechanical 
Equipment 


Fixing,  servicing,  aligning,  setting  up,  adjusting,  and 
testing  machines,  devices,  moving  parts,  and 
equipment  that  operate  primarily  on  the  basis  of 
mechanical  (not  electronic)  principles. 


What  level  of  this 


Level 

activity  is  needed  to  perform  this  job? 


Performing  complex  or  non-routine 

KU- 

repair,  maintenance,  or  adjustment  of 

rts 

mechanical  equipment,  often  involving 

(6)- 

overhauls  or  rebuilding. 

©- 

- *— 

Performing  straightforward  repair, 

®- 

maintenance,  or  adjustment  of 

mechanical  equipment  using 

©- 

established,  easy  to  understand 

^ 

procedures. 

Rebuilding  a high-performance  engine. 


Removing  and  replacing  broken  parts  in  an 
automobile  transmission. 


Adjusting  a grandfather  clock. 


Making  routine  preventive  maintenance  to 
a door  lock. 


Making  simple , external  adjustments  to  a 
j_  door  hinge  with  ordinary  Hand  tools. 

^R)  Not  relevant  at  all  for  performance  on  this  job 


Not 

Important 


Importance 

How  important  is  this  activity  to  performance  on  this  job? 

Somewhat  Very 

Important  Important  Important 


Extremely 

Important 


© 

© 

© 

© 

© 

Frequency 

How  often  is  this  activity  performed  on  this  job? 

Once  per 
year  or  less 

1 

More  than 
once  per  year 

i 

More  than  More  than 

once  per  month  once  per  week 

Several  times 
Daily  per  day 

Hourly  or 
more  often 

© 

© 

® © 

j | 

© © 

® 

Table  6-1 

Descriptive  Statistics  Across  All  Occupations  and  Reliability  Estimates  for  Rated  Differences 
Between  Occupations:  Generalized  Work  Activities 


Variable 


Level Importance Frequency 


Descriptor 

M 

SD 

SEM 

a b 

Ik 

M 

SD 

SEM 

Ik 

M 

SD 

SEM 

Ik 

1.  Getting  Information 

3.47 

1.35 

.55 

.83 

3.15 

0.70 

.35 

0.75 

4.06 

0.88 

.53 

.64 

2.  Identifying  Objects 

2.92 

1.28 

.57 

.80 

2.75 

0.73 

.37 

0.75 

3.79 

0.87 

.58 

.56 

3.  Monitoring  Processes 

2.77 

1.23 

. .56 

.79 

2.65 

0.71 

.36 

0.75 

3.79 

1.04 

.60 

.67 

4.  Inspecting  Equipment 

1.91 

1.10 

.52 

.77 

2.31 

0.69 

.32 

0.78 

2.99 

0.82 

.47 

.67 

5.  Estimating  Characteristics  1.76 

0.83 

.50 

.64 

2.09 

0.50 

.33 

0.57 

2.73 

0.72 

.50 

.53 

6.  Judging  the  Qualities 

2.58 

1.30 

.55 

.82 

2.47 

0.72 

.34 

0.78 

3.26 

0.96 

.54 

.68 

7.  Evaluating  Information 

2.72 

1.27 

.52 

.83 

2.82 

0.72 

.35 

0.77 

3.63 

0.98 

.50 

.74 

8.  Processing  Information 

2.54 

1.27 

.62 

.76 

2.54 

0.74 

.36 

0.77 

3.36 

1.07 

.58 

.71 

9.  Analyzing  Data 

2.37 

1.54 

.58 

.86 

2.41 

0.77 

.33 

0.82 

2.98 

0.90 

.44 

.76 

10.  Making  Decisions 

3.12 

1.34 

.54 

.84 

2.90 

0.73 

.32 

0.80 

3.79 

0.88 

.48 

.70 

11.  Thinking  Creatively 

2.78 

1.33 

.61 

.79 

2.51 

0.72 

.33 

0.78 

3.17 

0.92 

.48 

.73 

12.  Using  Job  Knowledge 

3.80 

1.22 

.61 

.75 

3.19 

0.68 

.36 

0.72 

3.59 

0.70 

.51 

.47 

13.  Developing  Objectives 

2.19 

1.28 

.48 

.86 

2.26 

0.75 

.27 

0.87 

2.37 

0.62 

.36 

.65 

14.  Scheduling  Work 

2.44 

1.18 

.52 

.81 

2.46 

0.67 

.31 

0.79 

3.06 

1.00 

.46 

.79 

15.  Organizing  and  Planning 

3.80 

1.18 

.52 

.80 

3.26 

0.73 

.32 

0.80 

4.30 

0.86 

.46 

.72 

16.  Perform.  Phys.  Work  Tasks2.70 

1.35 

.54 

.84 

2.51 

0.76 

.32 

0.82 

4.26 

1.11 

.68 

.62 

17.  Handling  Objects 

3.40 

0.91 

.64 

.51 

2.86 

0.54 

.36 

0.56 

4.50 

0.84 

.62 

.46 

18.  Controlling  Machines 

1.70 

1.54 

.62 

.84 

2.17 

0.83 

.37 

0.80 

3.07 

1.39 

.66 

.77 

19.  Interacting  with  Computers  2.01 

1.39 

.54 

.85 

2.50 

0.86 

.31 

0.87 

3.48 

1.47 

.50 

.89 

20.  Operating  Vehicles 

1.53 

1.54 

.44 

.92 

1.96 

0.95 

.26 

0.92 

2.54 

1.51 

.41 

.92 

21.  Specifying  Equipment 

0.87 

0.92 

.40 

.81 

1.48 

0.55 

.24 

0.81 

1.59 

0.65 

.30 

.78 

22.  Implementing  Ideas 

2.02 

1.00 

.54 

.71 

2.25 

0.62 

.33 

0.71 

2.82 

0.78 

.49 

.61 

23.  Repairing,  Mechanical 

1.11 

1.25 

.47 

.86 

1.63 

0.74 

.27 

0.86 

1.85 

0.93 

.39 

.82 

24.  Repairing,  Electronic 

1.04 

1.25 

.55 

.80 

1.62 

0.69 

.32 

0.78 

1.91 

0.82 

.46 

.68 

25.  Documenting  Information 

2.45 

1.05 

,61 

.67 

2.67 

0.65 

.38 

0.65 

3.63 

0.93 

.57 

.63 

26.  Interpreting  Information 

2.32 

0.93 

.54 

.66 

2.48 

0.57 

.35 

0.63 

3.40 

0.92 

.56 

.64 

27.  Communicating,  Internal 

4.14 

1.14 

.51 

.80 

3.54 

0.57 

.30 

0.73 

4.78 

0.66 

.51 

.41 

28.  Communicating,  External 

3.09 

1.35 

.56 

.83 

2.88 

0.83 

.34 

0.84 

3.62 

1.13 

.54 

.77 

29.  Establishing  Relationships  4.48 

0.93 

.48 

.73 

3.76 

0.63 

.33 

0.73 

5.18 

0.78 

.44 

.68 

30.  Assisting  Others 

3.08 

1.05 

.56 

.72 

3.11 

0.58 

.37 

0.59 

4.16 

0.86 

.53 

.63 

31.  Selling  or  Influencing 

2.07 

1.18 

.52 

.81 

2.31 

0.69 

.32 

0.79 

2.94 

0.92 

.49 

.71 

32.  Resolving  Conflicts 

2.82 

1.25 

.48 

.85 

2.77 

0.75 

.33 

0.81 

3.19 

0.86 

.42 

.77 

33.  Working  with  the  Public 

2.58 

1.49 

.65 

.81 

2.65 

0.90 

.43 

0.77 

3.41 

1.35 

.65 

.77 

34.  Coordinating  Others’  Work2.43 

1.24 

.47 

.86 

2.57 

0.65 

.30 

0.78 

3.31 

0.77 

.47 

.63 

35.  Developing  Teams 

2.46 

1.09 

.49 

.80 

2.56 

0.60 

.29 

0.76 

3.18 

0.67 

.43 

.60 

36.  Teaching  Others 

2.48 

0S7 

.48 

-76 

2.70 

0.55  -33 

0.64 

3.11 

0.S3 

.45 

-70 

Table  6-1  (continued) 

Descriptive  Statistics  Across  All  Occupations  and  Reliability  Estimates  for  Rated  Differences 
Between  Occupations:  Generalized  Work  Activities 


Variable 


Level Importance Frequency 


Descriptor 

M 

SD 

SEMa 

1 r b 
. Of 

M 

SD 

SEM 

Ik 

M 

SD 

SEM 

Ik 

37.  Directing  Subordinates 

2.01 

1.34 

.41 

.91 

2.30 

0.73 

.26 

0.87 

2.69 

0.89 

.36 

.84 

38.  Developing  Others 

2.58 

1.07 

.45 

.82 

2.66 

0.61 

.31 

.74 

3.15 

0.73 

.39 

.71 

39.  Providing  Consultation 

2.17 

1.15 

.49 

.82 

2.28 

0.61 

.29 

.78 

2.71 

0.69 

.40 

.65 

40.  Performing  Admin.  Tasks 

2.38 

1.18 

.53 

.80 

2.59 

0.64 

.31 

.77 

3.52 

1.00 

.49 

.76 

41.  Staffing  Org.  Units 

1.19 

1.12 

.39 

.88 

1.73 

0.66 

.22 

.89 

1.57 

0.46 

.29 

.61 

42.  Monitoring  Resources 

1.41 

1.17 

.47 

.84 

1.85 

0.72 

.29 

.84 

2.13 

0.71 

.41 

.67 

Note.  Statistics  are  based  on  35  occupations  with  Generalized  Work  Activities  questionnaire 
responses  from  at  least  four  incumbents  (mean  number  of  incumbents  = 18.91,  median  = 13, 
harmonic  mean  = 9.68). 

a This  estimate  of  the  standard  error  of  measurement  was  calculated  as  SEM  = SD*  . 

b This  estimate  of  reliability  was  obtained  by  calculating  the  intraclass  correlation  for  k ratings 
across  occupations:  ICC(lJc)  = IBMS-WMS1/BMS  (Shrout  & Fleiss,  1979),  where  k is  the 
harmonic  mean  of  the  number  of  ratings  provided  on  each  occupation. 


Table  6-2 

Reliability  of  Rated  Differences  Between  Occupations  Considering  Varying  Numbers  of  Raters: 
Generalized  Work  Activities 


Variable 


Level Importance Frequency 


Descriptor 

Iia 

l30b 

II 

130 

II 

130 

1.  Getting  Information 

34 

94 

24 

90 

16 

85 

2.  Identifying  Objects 

30 

93 

24 

90 

12 

80 

3.  Monitoring  Processes 

28 

92 

24 

90 

17 

86 

4.  Inspecting  Equipment 

26 

91 

27 

92 

18 

87 

5.  Estimating  Characteristics 

15 

85 

12 

80 

10 

78 

6.  Judging  the  Qualities 

33 

94 

26 

91 

18 

87 

7.  Evaluating  Information 

33 

94 

25 

91 

23 

90 

8.  Processing  Information 

25 

91 

25 

91 

20 

88 

9.  Analyzing  Data 

39 

95 

31 

93 

25 

91 

10.  Making  Decisions 

35 

94 

29 

93 

19 

88 

11.  Thinking  Creatively 

28 

92 

27 

92 

22 

89 

12.  Using  Job  Knowledge 

23 

90 

21 

89 

08 

73 

13.  Developing  Objectives 

39 

95 

40 

95 

16 

85 

14.  Scheduling  Work 

30 

93 

29 

92 

28 

92 

15.  Organizing  and  Planning 

29 

93 

30 

93 

21 

89 

16.  Performing  Physical  Work  Tasks 

35 

94 

32 

93 

15 

84 

17.  Handling  Objects 

10 

76 

12 

80 

08 

72 

18.  Controlling  Machines 

35 

94 

29 

92 

26 

91 

19.  Interacting  with  Computers 

36 

94 

42 

96 

45 

96 

20.  Operating  Vehicles 

54 

97 

55 

97 

56 

97 

21.  Specifying  Equipment 

30 

93 

31 

93 

27 

92 

22.  Implementing  Ideas 

20 

88 

20 

88 

14 

83 

23.  Repairing,  Mechanical 

39 

95 

39 

95 

32 

93 

24.  Repairing,  Electronic 

30 

93 

27 

92 

18 

87 

25.  Documenting  Information 

17 

86 

16 

85 

15 

84 

26.  Interpreting  Information 

17 

86 

15 

84 

15 

85 

27.  Communicating,  Internal 

29 

92 

22 

90 

07 

69 

28.  Communicating,  External 

33 

94 

34 

94 

26 

91 

29.  Establishing  Relationships 

22 

89 

22 

89 

18 

87 

30.  Assisting  Others 

21 

89 

13 

82 

15 

84 

31.  Selling  or  Influencing 

30 

93 

27 

92 

20 

89 

32.  Resolving  Conflicts 

37 

95 

31 

93 

25 

91 

33.  Working  with  the  Public 

30 

93 

26 

91 

26 

91 

34.  Coordinating  Others’  Work 

38 

95 

27 

92 

15 

84 

35.  Developing  Teams 

29 

93 

25 

91 

13 

82 

36.  Teaching  Others 

25 

91 

16 

85 

20 

88 

37.  Directing  Subordinates 

50 

97 

42 

96 

35 

94 

545 


Table  6-2  (continued) 


iStCiiauiiiiY  oi  Kaiea  jLiirierences  t 
Generalized  Work  Activities 

Descriptor 

between  uc 

:cut>ations  C 

onsidenne  Varvins:  Nun 
Variable 

ibers  of  Rate 

srs: 

Level 

ImDortance 

Freauencv 

iia 

E30b 

II 

Do 

II 

130 

38.  Developing  Others 

32 

93 

23 

90 

20 

88 

39.  Providing  Consultation 

32 

93 

27 

92 

16 

85 

40.  Performing  Admin.  Tasks 

29 

93 

25 

91 

24 

91 

41.  Staffing  Org.  Units 

43 

96 

44 

96 

14 

83 

42.  Monitoring  Resources 

35 

. 94 

34 

94 

17 

86 

Note.  Reliability  estimates  are  based  on  35  occupations  with  Generalized  Work  Activities  questionnaire 
responses  from  at  least  4 incumbents  (mean  number  of  incumbents  = 18.9,  median  = 13,  harmonic  mean 
= 9.68).  Decimals  are  omitted. 

a Single  rater  estimates  of  reliability  were  obtained  by  calculating  the  intraclass  correlation  for  single 
judge  ratings  across  occupations:  ICC(1,1)  = rBMS-WMSl/BMS  + (k-lVWMS  (Shrout  & Fleiss,  1979), 
where  k is  the  harmonic  mean  of  the  number  of  ratings  provided  on  each  occupation. 

Estimates  of  reliability  for  30  raters  were  obtained  by  applying  the  Spearman-Brown  correction  formula 
to  the  single  rater  reliability  estimates. 
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Table  6-3 

Reliability  of  Rated  Differences  Between  Occupations  Considering  Various  Recoding  Schemes: 
Generalized  Work  Activities 


Type  of  Scale  and  Recoding  Scheme  Applied 


Descriptor 

Level 

Importance 

Freauencv 

la 

Eb 

Ec 

Ea 

Eb 

la 

lb 

1.  Getting  Information 

83 

83 

67 

75 

75 

64 

64 

2.  Identifying  Objects 

80 

80 

68 

75 

74 

56 

56 

3.  Monitoring  Processes 

79 

79 

70 

75 

75 

67 

67 

4.  Inspecting  Equipment 

77 

77 

69 

78 

77 

67 

67 

5.  Estimating  Characteristics 

64 

64 

48 

57 

55 

53 

53 

6.  Judging  the  Qualities 

82 

82 

67 

78 

77 

68 

68 

7.  Evaluating  Information 

83 

83 

79 

77 

79 

74 

74 

8.  Processing  Information 

76 

76 

76 

77 

77 

71 

71 

9.  Analyzing  Data 

86 

86 

81 

82 

83 

76 

76 

10.  Making  Decisions 

84 

84 

70 

80 

79 

70 

70 

11.  Thinking  Creatively 

79 

79 

77 

78 

80 

'73 

73 

12.  Using  Job  Knowledge 

75 

75 

70 

72 

74 

47 

47 

13.  Developing  Objectives 

86 

86 

81 

87 

86 

65 

65 

14.  Scheduling  Work 

81 

81 

82 

79 

81 

79 

79 

15.  Organizing  and  Planning 

80 

80 

78 

80 

82 

72 

72 

16.  Perform.  Phys.  Work  Tasks 

84 

84 

61 

82 

80 

62 

62 

17.  Handling  Objects 

51 

51 

46 

56 

54 

46 

46 

18.  Controlling  Machines 

84 

84 

72 

80 

79 

77 

77 

19.  Interacting  with  Computers 

85 

85 

87 

87 

88 

89 

89 

20.  Operating  Vehicles 

92 

92 

88 

92 

92 

92 

92 

21.  Specifying  Equipment 

81 

81 

76 

81 

81 

78 

78 

22.  Implementing  Ideas 

71 

71 

68 

71 

71 

61 

61 

23.  Repairing,  Mechanical 

86 

86 

78 

86 

85 

82 

82 

24.  Repairing,  Electronic 

80 

80 

61 

78 

75 

68 

68 

25.  Documenting  Information 

67 

67 

64 

65 

66 

63 

63 

26.  Interpreting  Information 

66 

66 

58 

63 

63 

64 

64 

27.  Communicating,  Internal 

80 

80 

64 

73 

74 

41 

41 

28.  Communicating,  External 

83 

83 

81 

84 

84 

77 

77 

29.  Establishing  Relationships 

73 

73 

75 

73 

75 

68 

68 

30.  Assisting  Others 

72 

72 

33 

59 

56 

63 

63 

31.  Selling  or  Influencing 

81 

81 

60 

79 

76 

71 

71 

32.  Resolving  Conflicts 

85 

85 

77 

81 

81 

77 

77 

33.  Working  with  the  Public 

81 

81 

70 

77 

77 

77 

77 

34.  Coordinating  Others’  Work 

86 

86 

73 

78 

78 

63 

63 

35.  Developing  Teams 

80 

80 

70 

76 

77 

60 

60 

36.  Teaching  Others 

76 

76 

54 

64 

62 

70 

70 

37.  Directing  Subordinates 

91 

91 

81 

87 

87 

84 

84 

547 


Table  6-3  (continued) 

Reliability  of  Rated  Differences  Between  Occupations  Considering  Various  Recoding  Schemes: 
Generalized  Work  Activities 


Type  of  Scale  and  Recoding  Scheme  Applied 


Level 


Importance 


Descriptor 

la 

lb 

Ic 

la 

lb 

la 

lb 

38.  Developing  Others 

82 

82 

55 

74 

73 

71 

71 

39.  Providing  Consultation 

82 

82 

64 

78 

77 

65 

65 

40.  Performing  Admin.  Tasks 

80 

80 

75 

77 

77 

76 

76 

41.  Staffing  Org.  Units 

88 

88 

85 

89 

89 

61 

61 

42.  Monitoring  Resources 

84 

84 

79 

84 

83 

67 

67 

$°*e‘  Reliability  estimates  are  based  on  35  occupations  with  Generalized  Work  Activities  questionnaire 
responses  from  at  least  4 incumbents  (mean  number  of  incumbents  = 18.9,  median  = 13,  harmonic  mean 
= 9.68).  Reliability  estimates  stipulated  as  i*  were  calculated  using  the  full  eight  point  scale  for  level,  and 
retaining  all  of  the  data  for  the  importance  and  frequency  scales.  Reliability  estimates  stipulated  as  it, 
were  calculated  using  a reduced  seven  point  scale  for  level,  and  excluding  the  data  for  the  importance 
and  frequency  scales  where  the  rater  marked  “NR”  on  the  level  scale.  Reliability  estimates  stipulated  as  £. 
were  calculated  using  a binary  coded  scale  for  level  (relevant/not  relevant).  Decimals  are  omitted. 


Table  6-4a 

Analysis  of  Variance  for  Descriptor.  Occupation,  and  Relevant  Interactions  as  Sources  of  Variation  on 
the  Level  Scale:  Generalized  Work  Activities 


Source  of  Variation 

SS 

df 

MS 

F 

Occupations 

11915.96 

34 

350.47 

6.06* 

S (Occupations) 

36274.33 

627 

57.85 

Descriptor 

9025.69 

41 

220.14 

85.55* 

Descriptor  x Occupations 

17833.73 

1394 

12.79 

4.97* 

Descriptor  x S(Occupations) 

66151.48 

25707 

2.57 

Note.  Occupations  are  treated  as  random,  between-subjects  effects,  while  descriptors  are  treated  as  fixed, 
within-subjects  effects. 

*£<.05 


Table  6-4b 

Analysis  of  Variance  for  Descriptor,  Occupation,  and  Relevant  Interactions  as  Sources  of  Variation  nn 
the  Importance  Scale:  Generalized  Work  Activities 


Source  of  Variation 

SS 

df 

MS 

F 

Occupations 

3087.70 

34 

90.81 

5.1* 

S(Occupations) 

11154.37 

627 

17.79 

Descriptor 

3309.13 

41 

80.71 

76.48* 

Descriptor  x Occupations 

6781.85 

1394 

4.87 

4.61* 

Descriptor  x S(Occupations) 

27127.57 

25707 

1.06 

£fo-*e-  Occupations  are  treated  as  random,  between-subjects  effects,  while  descriptors  are  treated  as  fixed, 
within-subjects  effects. 

*P<.05 


Table  6-4c 

Analysis  of  Variance  for  Descriptor.  Occupation,  and  Relevant  Interactions  as  Sources  of  Variation  on 
the  Frequency  Scale:  Generalized  Work  Activities 


Source  of  Variation 

SS 

df 

MS 

F 

Occupations 

3803.95 

34 

111.88 

2.98* 

S(Occupations) 

23560.76 

627 

37.58 

Descriptor 

8849.46 

41 

215.84 

94.08* 

Descriptor  x Occupations 

12494.13 

1394 

8.96 

3.91* 

Descriptor  x S(Occupations) 

58974.69 

25707 

2.29 

Note.  Occupations  are  treated  as  random,  between-subjects  effects,  while  descriptors  are  treated  as  fixed, 
within-subjects  effects. 

*£<.05 


Table  6-5 

Interrater  Agreement  Coefficients  for  Each  Scale  Type:  Generalized  Work  Activities 


Scale  Type 

Number  of  Raters  on  Each  Variable 

Ik 

II 

130 

Level 

80 

29 

92 

Importance 

78 

27 

92 

Frequency 

74 

23 

90 

Note.  Interrater  agreement  coefficient  estimates  are  based  on  35  occupations  with  Generalized  Work 
Activities  questionnaire  responses  from  at  least  4 incumbents  (mean  number  of  incumbents  = 18.91, 
median  = 13,  harmonic  mean  = 9.68).  Full  sample  interrater  agreement  coefficients  (i*  ) were  obtained  by 
considering  the  “Descriptor  x Occupations”  terms  from  Tables  6-4a,  6-4b,  and  6-4c  as  true  variance. 
Error  variance  was  defined  as  the  “Descriptor  x S(Occupations)”  term.  Estimates  of  reliability  for  1 and 
30  raters  were  obtained  by  applying  the  Spearman-Brown  correction  formula  to  the  k rater  reliability 
estimates,  where  k is  the  harmonic  mean  of  the  number  of  raters  for  each  occupation.  Decimals  are 
omitted. 


0 

ERIC 


55.2 


Table  6-6a 

Analysis  of  Variance  for  Aggregate  Descriptor.  Occupation,  and  Relevant  Interactions  as  Sources  of 
Variation  on  the  Level  Scale:  Generalized  Work  Activities 


Source  of  Variation 

SS 

df 

MS 

F 

Occupations 

2647.17 

34 

77.86 

6.23* 

S(Occupations) 

7842.03 

627 

12.51 

Aggregate 

894.05 

8 

111.76 

106.12* 

Aggregate  x Occupations 

2030.55 

272 

7.47 

7.09* 

Aggregate  x S(Occupations) 

5282.65 

5016 

1.05 

Note.  Occupations  are  treated  as  random,  between-subjects  effects,  while  aggregate  descriptors  are 
treated  as  fixed,  within-subjects  effects. 

*£<.05 


553 


? 


Table  6-6b 


Mansis  of  Variance  for  Aggregate  Descriptor.  Occupation,  and  Relevant  Interactions  as  Sources  nf 
Variation  on  the  Importance  Scale:  Generalized  Work  Activities 


Source  of  Variation 

SS 

df 

MS 

F 

Occupations 

668.82 

34 

19.67 

5.01* 

S(Occupations) 

2461.78 

627 

3.92 

Aggregate 

296.37 

8 

37.04 

93.09* 

Aggregate  x Occupations 

698.23 

272 

2.57 

6.45* 

Aggregate  x S(Occupations) 

1996.17 

5016 

.40 

Note,  Occupations  are  treated  as  random,  between-subjects  effects,  while  aggregate  descriptors  are 
treated  as  fixed,  within-subjects  effects. 

*P<.05 


Table  6-6c 

Analysis  of  Variance  for  Aggregate  Descriptor.  Occupation,  and  Relevant  Interactions  as  Sources  of 
Variation  on  the  Frequency  Scale:  Generalized  Work  Activities 


Source  of  Variation 

SS 

df 

MS 

F 

Occupations 

821.87 

34 

24.17 

2.88* 

S(Occupations) 

5268.78 

627 

8.40 

Aggregate 

743.07 

8 

92.88 

105.36* 

Aggregate  x Occupations 

1093.33 

272 

4.02 

4.56* 

Aggregate  x S(Occupations) 

4422.24 

5016 

.88 

Note.  Occupations  are  treated  as  random,  between-subjects  effects,  while  aggregate  descriptors  are 
treated  as  fixed,  within-subjects  effects. 

*p<.05 
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Table  6-7 

Interrater  Agreement  Coefficients  for  Aggregate  Descriptors  for  Each  Scale  Type:  Generalised  WnrV 
Activities 


Scale  Type 

Number  of  Raters  on  Each  Variable 

Ik 

lx 

Do 

Level 

86 

39 

95 

Importance 

84 

36 

94 

Frequency 

78 

27 

92 

No^.  Interrater  agreement  coefficient  estimates  are  based  on  35  occupations  with  Generalized  Work 
Activities  questionnaire  responses  from  at  least  4 incumbents  (mean  number  of  incumbents  = 18.91, 
median  = 13,  harmonic  mean  = 9.68).  Full  sample  interrater  agreement  coefficients  (r*  ) were  obtained  by 
considering  the  “Aggregate  x Occupations”  terms  from  Tables  6-6a,  6-6b,  and  6-6c  as  true  variance. 

Error  variance  was  defined  as  the  “Aggregate  x S(Occupations)”  term.  Estimates  of  reliability  for  1 and 
30  raters  were  obtained  by  applying  the  Spearman-Brown  correction  formula  to  the  k rater  reliability 
estimates,  where  k is  the  harmonic  mean  of  the  number  of  raters  for  each  occupation.  Decimals  are 
omitted. 


Table  6-8 

Means  and  Standard  Deviations  of  Correlations  Between  Level.  Importance,  and  Frequency  Scales 
Across  Occupations  and  Descriptors:  Generalized  Work  Activities 


Level Importance Frequency 


Scale 

na 

M 

SD 

n 

M 

SD 

n 

M 

SD 

Level 

— 

— 

— 

35 

.93 

.11 

35 

.88 

.08 

Importance 

42 

.92 

.05 

— 

— 

— 

35 

.91 

.06 

Frequency 

42 

.82 

.07 

42 

.89 

.06 

— 

— 

— 

Note.  All  correlations  were  calculated  based  on  the  mean  of  ratings  assigned  by  raters  for  a given 
occupation,  descriptor,  and  scale.  Level-importance  means  above  the  diagonal  were  calculated  by  taking 
the  level  scale  means  on  a given  occupation  for  all  descriptors,  correlating  them  with  importance  scale 
means,  for  that  occupation,  and  then  averaging  them  with  the  correlations  for  other  occupations.  Level- 
importance  means  below  the  diagonal  were  calculated  by  taking  the  level  scale  means  on  a given 
descriptor  for  all  occupations,  correlating  them  with  importance  scale  means  for  that  descriptor,  and 
averaging  them  with  correlations  for  other  descriptors.  Other  means  in  the  table  were  calculated  in  a 
similar  manner. 

a Number  of  correlations  averaged,  not  number  of  observations  on  which  correlations  were  calculated. 


Table  6-9a 

Intercorrelations  of  Descriptors  for  the  Level  Scale  (Occupation-Level  Data):  Generalized  Work  Activities 
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Table  6-9a  (continued) 

IhtercOrrelations  of  Descriptors  for  the  Level  Scale  (Occupation-Level  Data):  Generalized  Work  Activities 
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Table  6-9b 

Intercbrrelations  of  Descriptors  for  the  Importance  Scale  (Occupation-Level  Data):  Generalized  Work  Activities 
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Table  6-9b  (continued) 

Intercorrelations  of  Descriptors  for  the  Importance  Scale  f Occupation -Level  Data):  Generalized  Work  Activities 
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Table  6-9c  . 

Intercorrelations  of  Descriptors  for  the  Frequency  Scale  (Occupation-Level  Data):  Generalized  Work  Activities 
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5 ^correlations  of  Descriptors  for  the  Importance  Scale  (Incumbent-Level  Data):  Generalized  Work  Activities 
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Table  6-1 1 

Principal  Components  Analysis  Pattern  Matrix  for  the  Incumbent  Level  Scale:  Generalized  Work 
Activities 


Factor 


Descriptor 

FI 

F2 

F3 

Communality 

1.  Getting  Information 

.82 

.30 

-.17 

.78 

2.  Identifying  Objects 

.81 

.44 

.05 

.85 

3.  Monitoring  Processes 

.69 

.47 

.33 

.81 

4.  Inspecting  Equipment 

.24 

.00 

.90 

.86 

5.  Estimating  Characteristics 

.70 

.23 

.46 

.76 

6.  Judging  the  Qualities 

.65 

.61 

-.12 

.81 

7.  Evaluating  Information 

.84 

.40 

.02 

.87 

8.  Processing  Information 

.78 

.12 

-.40 

.79 

9.  Analyzing  Data 

.87 

.21 

.20 

.83 

10.  Making  Decisions 

.82 

.50 

.06 

.94 

11.  Thinking  Creatively 

.86 

.20 

.14 

.80 

12.  Using  Job  Knowledge 

.80 

.33 

.15 

.76 

13.  Developing  Objectives 

.61 

.65 

-.16 

.82 

14.  Scheduling  Work 

.49 

.54 

-.11 

.55 

15.  Organizing  and  Planning 

.73 

.45 

-.15 

.75 

16.  Performing  Phys.  Work  Tasks 

-.39 

.22 

.68 

.67 

17.  Handling  Objects 

-.06 

-.35 

.51 

.38 

18.  Controlling  Machines 

-.15 

-.21 

.82 

.75 

19.  Interacting  with  Computers 

.82 

-.27 

-.05 

.75 

20.  Operating  Vehicles 

-.24 

.14 

.63 

.48 

21.  Specifying  Equipment 

.41 

-.12 

.56 

.49 

22.  Implementing  Ideas 

.81 

.14 

.28 

.75 

23.  Repairing,  Mechanical 

.15 

-.18 

.89 

.85 

24.  Repairing,  Electronic 

.27 

-.21 

.82 

.80 

25.  Documenting  Information 

.52 

.32 

-.15 

.40 

26.  Interpreting  Information 

.75 

.40 

-.14 

.75 

27.  Communicating,  Internal 

.75 

.43 

-.05 

75 

28.  Communicating,  External 

.57 

.53 

-.31 

• / J 

.70 

29.  Establishing  Relationships 

.35 

.63 

-.10 

.53 

30.  Assisting  Others 

-.13 

.74 

-.06 

.57 

31.  Selling  or  Influencing 

.35 

.57 

-.17 

.47 

32.  Resolving  Conflicts 

.29 

.78 

-.12 

.70 

33.  Working  with  the  Public 

-.15 

.66 

-.22 

.50 

34.  Coordinating  Others’  Work 

.44 

.80 

-.01 

.84 

35.  Developing  Teams 

.50 

.74 

-.09 

.80 

36.  Teaching  Others 

.42 

.61 

.10 

.56 

37.  Directing  Subordinates 

.34 

.88 

.01 

.89 

38.  Developing  Others 

.36 

.85 

-.01 

* .84 

Table  6-1 1 (continued) 


Activities 

Descriptor 

Factor 

FI 

F2 

F3 

Communality 

39.  Providing  Consultation 

•73 

.54 

.02 

.83 

40.  Performing  Admin.  Tasks 

.47 

.61 

-.35 

.72 

41.  Staffing  Org.  Units 

.42 

.54 

.13 

.48 

42.  Monitoring  Resources 

.50 

.48 

.16 

.51 

Percent  of  Variance 

48 

14 

8 

Eigenvalue 

20.23 

6.08 

3.41 

Note.  N = 35.  The  correlation  matrix  was  based  on  means  calculated  at  the  occupation  level.  FI  = 
Working  with  Information,  F2  = Working  with  and  Directing  the  Activities  of  Others,  and  F3  = Manual 
and  Physical  Activities:  Performing  Repair  and  Other  Physical  Work.  These  loadings  are  based  on  an 
orthogonal  varimax  rotation. 
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Table  6- 12b  (continued) 

Descriptor  Means  and  Standard  Deviations  on  the  Importance  Scale  on  Six  Example  Occupations:  Generalized  Work  Activities 
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Descriptor.Means  and  Standard  Deviations  on  the  Frequency  Scale  on  Six  Example  Occupations:  Generalized  Work  Activities 
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Table  6- 12c  (continued) 

Descriptor  Means  and  Standard  Deviations  on  the  Frequency  Scale  on  Six  Example  Occupations:  Generalized  Work  Activities 
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Table  6-13 

Rotated  Correlations  Between  Discriminating  Variables  and  Canonical  Discriminant  Functions  for  th» 
Level  Scale:  Generalized  Work  Activities 


Functions 


Descriptor 

FI 

F2 

F3 

F4 

IF2 

'H2 

1.  Getting  Information 

.25 

.34 

.04 

-.01 

.17 

.24 

3.  Monitoring  Processes 

.32 

.15 

-.01 

.16 

.15 

.21 

2.  Identifying  Objects 

.26 

.23 

.06 

.07 

.13 

.21 

4.  Inspecting  Equipment 

.17 

-.07 

.04 

.41 

.20 

.19 

5.  Estimating  Characteristics 

.21 

.14 

.03 

.20 

.10 

.13 

6.  Judging  the  Qualities 

.37 

.11 

.03 

.01 

.15 

.24 

8.  Processing  Information 

.08 

.32 

-.08 

-.01 

.12 

.19 

7.  Evaluating  Information 

.21 

.35 

.04 

.08 

.17 

.24 

9.  Analyzing  Data 

.23 

.39 

.07 

.16 

.24 

.28 

11.  Thinking  Creatively 

.20 

.27 

-.02 

.10 

.12 

.21 

10.  Making  Decisions 

.31 

.30 

.14 

.10 

.21 

.25 

12.  Using  Job  Knowledge 

.20 

.23 

.05 

.11 

.11 

.18 

13.  Developing  Objectives 

.45 

.14 

-.00 

-.02 

.22 

.28 

14.  Scheduling  Work 

.32 

.14 

-.09 

-.05 

.13 

.22 

16.  Performing  Phys.  Work  Tasks 

-.04 

-.37 

.12 

.09 

.16 

.25 

15.  Organizing  and  Planning 

.20 

.30 

-.04 

-.03 

.13 

.21 

17.  Handling  Objects 

-.19 

-.05 

-.04 

.07 

.05 

.01 

18.  Controlling  Machines 

-.04 

-.19 

.05 

.38 

.19 

.25 

19.  Interacting  with  Computers 

-.10 

.48 

.01 

.02 

.24 

.26 

20.  Operating  Vehicles 

-.10 

-.10 

.61 

.35 

.51 

.04 

21.  Specifying  Equipment 

.07 

.06 

.03 

.39 

.16 

.22 

22.  Implementing  Ideas 

.21 

.18 

.03 

.20 

.12 

.16 

24.  Repairing,  Electronic 

-.01 

.03 

.01 

.45 

.20 

.22 

25.  Documenting  Information 

.04 

.10 

-.01 

-.04 

.01 

.14 

26.  Interpreting  Information 

.21 

.16 

-.02 

.05 

.07 

.14 

27.  Communicating,  Internal 

.16 

.35 

.07 

.02 

.15 

.21 

28.  Communicating,  External 

.10 

.34 

.19 

-.14 

.18 

.24 

29.  Establishing  Relationships 

.13 

.17 

.06 

-.11 

.06 

.17 

30.  Assisting  Others 

.13 

-.16 

.04 

-.14 

.06 

.16 

31.  Selling  or  Influencing 

.20 

.10 

.19 

-.10 

.10 

.22 

32.  Resolving  Conflicts 

.28 

.14 

.26. 

-.16 

.20 

.27 

33.  Working  with  the  Public 

.03 

-.07 

.23 

-.30 

.15 

.22 

34.  Coordinating  Others’  Work 

.49 

.07 

.01 

.02 

.24 

.27 

36.  Teaching  Others 

.34 

-.01 

-.06 

.06 

.12 

.18 

35.  Developing  Teams 

.38 

.13 

.06 

.01 

.16 

.21 

38.  Developing  Others 

.40 

.05 

.03 

-.04 

.16 

.23 

37.  Directing  Subordinates 

.61 

.09 

.10 

.04 

.39 

.37 

37.  Directing  Subordinates 

.61 

.09 

.10 

.04 

.39 

.37 

39.  Providing  Consultation 

.36 

.22 

.08 

.06 

.19 

.23 

40.  Performing  Admin.  Tasks 

.19 

.30 

.08 

-.06 

.14 

.21 

42.  Monitoring  Resources 

.43 

.16 

-.04 

.12 

.23 

.25 

41.  Staffing  Org.  Units 

.50 

.15 

.08 

.10 

29 

.31 

O 

ERJC 
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Table  6-13  (continued) 

Rotated  Correlations  Between  Discriminating  Variables  and  Canonical  Discriminant  Functions  for  the 
Level  Scale:  Generalized  Work  Activities 


Functions 


Descriptor 

FI 

F2 

F3 

F4 

IF2 

2 

11 

Rc 

.78 

.77 

.68 

.64 

Percent  of  Variance 

18 

17 

10 

8 

Eigenvalues 

1.56 

1.48 

.85 

.69 

Note.  Statistics  are  based  on  35  occupations  with  Generalized  Work  Activities  questionnaire 
responses  from  at  least  4 incumbents  (mean  number  of  incumbents  = 18.9,  median  =13,  harmonic  mean 
= 9.68).  FI  = Supervisory/Management;  F2  = Information  Processing;  F3  = Operating  Vehicles;  F4  = 
Manual/Physical  Activities/Repair  Related  Activities. 

IF2  = Sum  of  squared  rotated  standardized  discriminant  function  coefficients  across  four  functions. 

T)2  = Variance  in  Generalized  Work  Activity  Level  Scale  ratings  accounted  for  by  occupations. 


O 
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Table  6-14a 

Comparison  Between  Incumbent  and  Analyst  Descriptive  Statistics  Across  All  Occupations  anrf 
Reliability  Estimates  for  Rated  Differences  Between  Occupations  for  the  Level  Scale:  General  jypH  w^ri- 
Activities 


Incumbent Analyst 


Descriptor 

M 

SD 

Ik 

M 

SD 

IV 

t 

F 

Da 

d2 

1.  Getting  Information 

3.47 

1.35 

.83 

3.55 

1.14 

.95 

-0.57 

1.40 

.79 

0.67 

2.  Identifying  Objects 

2.92 

1.28 

.80 

3.21 

1.01 

.89 

-1.80 

1.60 

.68 

0.95 

3.  Monitoring  Processes 

2.77 

1.23 

.79 

2.95 

0.91 

.90 

-1.41 

1.85 

.77 

0.64 

4.  Inspecting  Equipment 

1.91 

1.10 

.77 

2.05 

0.99 

.92 

-1.12 

1.22 

.76 

0.54 

5.  Estimating  Characteristics 

1.76 

0.83 

.64 

2.32 

0.86 

.90 

-4.87* 

1.09 

.68 

0.76 

6.  Judging  the  Qualities 

2.58 

1.30 

.82 

2.75 

1.01 

.92 

-1.27 

1.68 

.80 

0.63 

7.  Evaluating  Information 

2.72 

1.27 

.83 

2.89 

0.87 

.91 

-1.07 

2.15* 

.69 

0.84 

8.  Processing  Information 

2.54 

1.27 

.76 

2.89 

1.12 

.94 

-2.83* 

1.29 

.82 

0.65 

9.  Analyzing  Data 

2.37 

1.54 

.86 

2.90 

1.20 

.96 

-2.67* 

1.67 

.68 

1.57 

10.  Making  Decisions 

3.12 

1.34 

.84 

2.76 

1.17 

.95 

2.65* 

1.31 

.80 

0.78 

11.  Thinking  Creatively 

2.78 

1.33 

.79 

2.28 

1.27 

.94 

3.18* 

1.09 

.74 

1.10 

12.  Using  Job  Knowledge 

3.80 

1.22 

.75 

2.96 

1.21 

.94 

4.86* 

1.01 

.64 

1.74 

13.  Developing  Objectives 

2.19 

1.28 

.86 

1.58 

1.37 

.94 

5.52* 

1.15 

.88 

0.79 

14.  Scheduling  Work 

2.44 

1.18 

.81 

1.91 

1.36 

.92 

3.30* 

1.34 

.73 

1.17 

15.  Organizing  and  Planning 

3.80 

1.18 

.80 

2.73 

1.16 

.93 

6.58* 

1.02 

.66 

2.04 

16.  Perform.  Phys.  Work  Tasks 

2.70 

1.35 

.84 

2.82 

1.21 

.93 

-0.75 

1.25 

.72 

0.93 

17.  Handling  Objects 

3.40 

0.91 

.51 

3.32 

0.89 

.87 

0.53 

1.04 

.47 

0.85 

18.  Controlling  Machines 

1.70 

1.54 

.84 

1.95 

1.15 

.92 

-1.78 

1.80 

.85 

0.74 

19.  Interacting  with  Computers 

2.01 

1.39 

.85 

1.89 

1.13 

.94 

0.76 

1.50 

.76 

0.82 

20.  Operating  Vehicles 

1.53 

1.54 

.92 

1.01 

1.24 

.95 

4.13* 

1.54 

.87 

0.82 

21.  Specifying  Equipment 

0.87 

0.92 

.81 

0.84 

1.19 

.95 

0.15 

1.69 

.63 

0.86 

22.  Implementing  Ideas 

2.02 

1.00 

.71 

2.37 

1.09 

.89 

-2.54* 

1.18 

.68 

0.80 

23.  Repairing,  Mechanical 

1.11 

1.25 

.86 

1.44 

1.33 

.95 

-2.42* 

1.13 

.80 

0.76 

24.  Repairing,  Electronic 

1.04 

1.25 

.80 

1.19 

0.64 

.75 

-0.85 

3.76* 

.58 

1.03 

25.  Documenting  Information 

2.45 

1.05 

.67 

2.80 

0.89 

.88 

-2.34* 

1.40 

.59 

0.88 

26.  Interpreting  Information 

2.32 

0.93 

.66 

2.29 

1.01 

.91 

0.17 

1.20 

.55 

0.84 

27.  Communicating,  Internal 

4.14 

1.14 

.80 

3.24 

1.20 

.94 

5.74* 

1.10 

.69 

1.64 

28.  Communicating,  External 

3.09 

1.35 

.83 

3.01 

1.37 

.93 

0.39 

1.03 

.68 

1.16 

29.  Establishing  Relationships 

4.48 

0.93 

.73 

3.44 

1.01 

.93 

6.21* 

1.17 

.47 

2.06 

30.  Assisting  Others 

3.08 

1.05 

.72 

2.18 

1.29 

.94 

5.35* 

1.51 

.65 

1.79 

31.  Selling  or  Influencing 

2.07 

1.18 

.81 

1.90 

1.32 

.93 

1.22 

1.25 

.79 

0.68 

32.  Resolving  Conflicts 

2.82 

1.25 

.85 

2.44 

1.41 

.95 

2.38* 

1.26 

.75 

1.02 

33.  Working  with  the  Public 

2.58 

1.49 

.81 

2.28 

1.41 

.91 

2.20* 

1.12 

.85 

0.73 

34.  Coordinating  Others’  Work 

2.43 

1.24 

.86 

2.31 

1.31 

.93 

0.68 

1.11 

.67 

1.05 

35.  Developing  Teams 

2.46 

1.09 

.80 

1.63 

1.44 

.93 

4.51* 

1.75 

.66 

1.83 

36.  Teaching  Others 

2.48 

0.97 

.76 

1.94 

1.32 

.93 

3.04* 

1.86 

.62 

1.34 

37.  Directing  Subordinates 

2.01 

1.34 

.91 

1.49 

1.68 

.96 

2.57* 

1.58 

.72 

1.61 

Table  6-14a  (continued) 

Comparison  Between  Incumbent  and  Analyst  Descriptive  Statistics  Across  AH  Occupations  and 


Activities 

Descriptor 

Incumbent 

Analyst 

M 

SD 

Ik 

M 

SD 

Ik 

t 

F 

lia 

d2 

38.  Developing  Others 

2.58 

1.07 

.82 

1.89 

1.38 

.93 

3.88* 

1.68 

.66 

1.55 

39.  Providing  Consultation 

2.17 

1.15 

.82 

2.40 

1.47 

.95 

-1.26 

1.64 

.69 

1.16 

40.  Performing  Admin.  Tasks 

2.38 

1.18 

.80 

2.55 

1.12 

.93 

-1.13 

1.10 

.70 

0.79 

41.  Staffing  Org.  Units 

1.19 

1.12 

.88 

0.96 

1.52 

.97 

1.41 

1.83 

.76 

1.00 

42.  Monitoring  Resources 

1.41 

1.17 

.84 

2.56 

1.06 

.87 

-7.54* 

1.21 

.67 

2.12 

Note.  Incumbent  statistics  are  based  on  35  occupations  with  Generalized  Work  Activities  questionnaire 
responses  from  at  least  four  incumbents  (mean  number  of  incumbents  = 18.91,  median  = 13,  harmonic 
mean  = 9.68).  Analyst  statistics  are  based  on  the  same  35  occupations  with  Generalized  Work  Activities 
questionnaire  responses  from  at  least  six  analysts  (mean  number  of  analysts  = 10.1 1,  median  = 12, 
harmonic  mean  = 8.37). 

The  estimate  of  reliability  was  obtained  by  calculating  the  intraclass  correlation  for  k ratings  across 
occupations:  ICC(l,k)  = fBMS-WMSl/BMS  (Shrout  & Fleiss,  1979),  where  k is  the  harmonic  mean  of 
the  number  of  ratings  provided  on  each  occupation. 

The  t statistic  tests  for  differences  in  the  incumbent  and  analyst  group  means. 

The  F statistic  tests  for  differences  in  the  incumbent  and  analyst  group  standard  deviations. 

The  I}*  correlation  indicates  the  degree  of  relationship  between  incumbent  and  analyst  mean  occupations 
ratings. 

The  d2  statistic  indicates  the  squared  differences  between  incumbent  and  analyst  mean  occupations 
ratings. 

* E < .05 
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Table  6-14b 

Comparison  Between  Incumbent  and  Analyst  Descriptive  Statistics  Across  All  Occupations  and 
Reliability  Estimates  for  Rated  Differences  Between  Occupations  for  the  Importance  Scale:  General^H 
Work  Activities 


Incumbent Analyst 


Descriptor 

M 

SD 

Ik 

M 

SD 

Ik 

t 

F 

ria 

d2 

1.  Getting  Information 

3.15 

0.70 

.75 

4.06 

0.48 

.77 

-11.49 

2.08 

.74 

1.05 

2.  Identifying  Objects 

2.75 

0.73 

.75 

3.51 

0.60 

.71 

-6.09 

1.49 

.40 

1.10 

3.  Monitoring  Processes 

2.65 

0.71 

.75 

3.46 

0.67 

.82 

-7.64 

1.12 

.58 

1.06 

4.  Inspecting  Equipment 

2.31 

0.69 

.78 

2.58 

0.99 

.91 

-2.36 

2.06 

.71 

0.54 

5.  Estimating  Characteristics 

2.09 

0.50 

.57 

2.93 

0.71 

.76 

-7.75 

2.04 

.47 

1.12 

6.  Judging  the  Qualities 

2.47 

0.72 

.78 

3.19 

0.64 

.77 

-6.87 

1.24 

.59 

0.89 

7.  Evaluating  Information 

2.82 

0.72 

.77 

3.45 

0.60 

.72 

-6.66 

1.43 

.65 

0.70 

8.  Processing  Information 

2.54 

0.74 

.77 

3.41 

0.81 

.85 

-8.91 

1.19 

.73 

1.09 

9.  Analyzing  Data 

2.41 

0.77 

.82 

3.29 

0.77 

.88 

-7.77 

1.01 

.62 

1.21 

10.  Making  Decisions 

2.90 

0.73 

.80 

3.15 

0.81 

.89 

-2.04 

1.24 

.58 

0.55 

11.  Thinking  Creatively 

2.51 

0.72 

.78 

2.43 

1.04 

.93 

0.53 

2.12 

.63 

0.65 

12.  Using  Job  Knowledge 

3.19 

0.68 

.72 

3.21 

0.76 

.83 

-0.14 

1.24 

.48 

0.53 

13.  Developing  Objectives 

2.26 

0.75 

.87 

2.13 

1.12 

.93 

1.30 

2.22* 

.90 

0.32 

14.  Scheduling  Work 

2.46 

0.67 

.79 

2.33 

0.98 

.90 

0.97 

2.13 

.59 

0.64 

15.  Organizing  and  Planning 

3.26 

0.73 

.80 

3.02 

0.83 

.86 

1.70 

1.29 

.43 

0.74 

16.  Perform.  Phys.  Work  Tasks 

2.51 

0.76 

.82 

3.04 

1.00 

.92 

-4.58 

1.74 

.73 

0.75 

17.  Handling  Objects 

2.86 

0.54 

.56 

3.45 

0.74 

.86 

-4.79 

1.84 

.39 

0.85 

18.  Controlling  Machines 

2.17 

0.83 

.80 

2.69 

1.09 

.92 

-4.45 

1.72 

.77 

0.75 

19.  Interacting  with  Computers 

2.50 

0.86 

.87 

2.68 

1.04 

.92 

-1.77 

1.45 

.79 

0.42 

20.  Operating  Vehicles 

1.96 

0.95 

.92 

1.88 

1.21 

.96 

0.87 

1.60 

.90 

0.29 

21.  Specifying  Equipment 

1.48 

0.55 

.81 

1.62 

0.90 

.94 

-1.17 

2.69* 

.64 

0.48 

22.  Implementing  Ideas 

2.25 

0.62 

.71 

3.04 

0.74 

.74 

-7.09 

1.41 

.54 

1.06 

23.  Repairing,  Mechanical 

1.63 

0.74 

.86 

1.93 

1.17 

.96 

-2.14 

2.54* 

.69 

0.79 

24.  Repairing,  Electronic 

1.62 

0.69 

.78 

1.63 

0.56 

.78 

-0.12 

1.50 

.64 

0.29 

25.  Documenting  Information 

2.67 

0.65 

.65 

3.69 

0.80 

.85 

-8.65 

1.52 

.55 

1.52 

26.  Interpreting  Information 

2.48 

0.57 

.63 

2.90 

0.94 

.89 

-3.58 

2.72* 

.67 

0.65 

27.  Communicating,  Internal 

3.54 

0.57 

.73 

3.53 

0.70 

.85 

0.05 

1.47 

.55 

0.37 

28.  Communicating,  External 

2.88 

0.83 

.84 

3.29 

1.15 

.93 

-2.59 

1.92 

.59 

1.04 

29.  Establishing  Relationships 

3.76 

0.63 

.73 

3.52 

0.84 

.91 

1.78 

1.81 

.44 

0.68 

30.  Assisting  Others 

3.11 

0.58 

.59 

2.52 

1.06 

.94 

4.06 

3.28* 

.58 

1.08 

31.  Selling  or  Influencing 

2.31 

0.69 

.79 

2.18 

1.11 

.94 

1.00 

2.55* 

.69 

0.65 

32.  Resolving  Conflicts 

2.77 

0.75 

.81 

2.69 

1.02 

.93 

0.66 

1.86 

.68 

0.55 

33.  Working  with  the  Public 

2.65 

0.90 

.77 

2.77 

1.20 

.91 

-1.00 

1.76 

.81 

0.50 

34.  Coordinating  Others’  Work 

2.57 

0.65 

.78 

2.59 

0.98 

.90 

-0.18 

2.29* 

.51 

0.71 

35.  Developing  Teams 

2.56 

0.60 

.76 

1.92 

1.05 

.93 

4.58 

3.05* 

.61 

1.08 

36.  Teaching  Others 

2.70 

0.55 

.64 

2.28 

1.22 

.95 

2.35 

4.97* 

.52 

1.23 

37.  Directing  Subordinates 

2.30 

0.73 

.87 

1.96 

1.22 

.96 

2.13 

2.80* 

.66 

0.94 
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Table  6- 14b  (continued) 

Comparison  Between  Incumbent  and  Analyst  Descriptive  Statistics  Across  All  Occupations  and 


Work  Activities 
Descriptor 

Incumbent 

Analyst 

M 

SD 

Ik 

M 

SD 

Ik 

t 

F 

La 

d2 

38.  Developing  Others 

2.66 

0.61 

.74 

2.06 

1.03 

.93 

3.64 

2.86* 

.39 

1.28 

39.  Providing  Consultation 

2.28 

0.61 

.78 

2.57 

1.08 

.93 

-1.94 

3.13* 

.56 

0.86 

40.  Performing  Admin.  Tasks 

2.59 

0.64 

.77 

3.27 

0.81 

.82 

-6.19 

1.64 

.61 

0.89 

41.  Staffing  Org.  Units 

1.73 

0.66 

.89 

1.58 

1.07 

.98 

1.22 

2.64* 

.75 

0.53 

42.  Monitoring  Resources 

1.85 

0.72 

.84 

3.03 

0.73 

.76 

-8.76 

1.02 

.39 

2.02 

Note.  Incumbent  statistics  are  based  on  35  occupations  with  Generalized  Work  Activities  questionnaire 
responses  from  at  least  four  incumbents  (mean  number  of  incumbents  =18.91,  median  = 13,  harmonic 
mean  = 9.68).  Analyst  statistics  are  based  on  the  same  35  occupations  with  Generalized  Work  Activities 
questionnaire  responses  from  at  least  six  analysts  (mean  number  of  analysts  = 10.1 1,  median  = 12, 
harmonic  mean  = 8.37). 

The  estimate  of  reliability  was  obtained  by  calculating  the  intraclass  correlation  for  k ratings  across 
occupations:  ICC(l.k)  = fBMS-WMSl/BMS  (Shrout  & Fleiss,  1979),  where  k is  the  harmonic  mean  of 
the  number  of  ratings  provided  on  each  occupation. 

The  t statistic  tests  for  differences  in  the  incumbent  and  analyst  group  means. 

The  F statistic  tests  for  differences  in  the  incumbent  and  analyst  group  standard  deviations. 

The  rja  correlation  indicates  the  degree  of  relationship  between  incumbent  and  analyst  mean  occupations 
ratings. 

The  d2  statistic  indicates  the  squared  differences  between  incumbent  and  analyst  mean  occupations 
ratings. 

* j>  < .05 
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Table  6-15 

Comparison  Between  Incumbent  and  Analyst  Descriptive  Statistics  Across  All  Occupations  for 
the  Level  Scale  Rescored  Dichotomouslv:  Generalized  Work  Activities 


Incumbent Analyst 


Descriptor 

M 

SD 

r a 

Ik 

M 

SD 

Ik 

t 

1 . Getting  Information 

.77 

.20 

.67 

1.00 

.00 

n/a 

-6.08* 

2.  Identifying  Objects 

.70 

.22 

.68 

0.98 

.05 

.21 

-6.56* 

3.  Monitoring  Processes 

.67 

.22 

.70 

0.98 

.04 

.00 

-7.33* 

4.  Inspecting  Equipment 

.54 

.21 

.69 

0.82 

.19 

.70 

-5.23* 

5.  Estimating  Characteristics 

.54 

.19 

.48 

0.91 

.12 

.44 

-8.71* 

6.  Judging  the  Qualities 

.63 

.22 

.67 

0.95 

.08 

.25 

- 7.23* 

7.  Evaluating  Information 

.71 

.23 

.79 

0.98 

.05 

.33 

-6.07* 

8.  Processing  Information 

.61 

.25 

.76 

0.97 

.09 

.62 

-7.16* 

9.  Analyzing  Data 

.61 

.26 

.81 

0.98 

.07 

.57 

-7.27* 

10.  Making  Decisions 

.71 

.20 

.70 

0.98 

.06 

.45 

-6.84* 

1 1 . Thinking  Creatively 

.69 

.24 

.77 

0.89 

.15 

.55 

-3.74* 

12.  Using  Job  Knowledge 

.81 

.20 

.70 

0.98 

.04 

.00 

-4.41* 

13.  Developing  Objectives 

.56 

.25 

.81 

0.61 

.33 

.83 

-0.64 

14.  Scheduling  Work 

.63 

.25 

.82 

0.69 

.28 

.78 

-0.85 

15.  Organizing  and  Planning 

.83 

.19 

.78 

0.98 

. .06 

.52 

-3.98* 

16.  Perform.  Phys.  Work  Tasks 

.71 

.23 

.61 

0.91 

.12 

.36 

-4.08* 

17.  Handling  Objects 

.75 

.16 

.46 

0.98 

.05 

.08 

- 7.26* 

18.  Controlling  Machines 

.46 

.27 

.72 

0.81 

.24 

.77 

-5.12* 

19.  Interacting  with  Computers 

.58 

.29 

.87 

0.82 

.28 

.88 

-3.15* 

20.  Operating  Vehicles 

.38 

.31 

.88 

0.39 

.35 

.90 

-0.11 

21.  Specifying  Equipment 

.23 

.22 

.76 

0.31 

.31 

.87 

-1.11* 

22.  Implementing  Ideas 

.57 

.22 

.68 

0.85 

.15 

.34 

-5.56* 

23.  Repairing,  Mechanical 

.28 

.25 

.78 

0.63 

.27 

.74 

-5.03* 

24.  Repairing,  Electronic 

.31 

.23 

.61 

0.64 

.21 

.52 

-5.60* 

25.  Documenting  Information 

.63 

.21 

.64 

0.96 

.11 

.74 

-7.36* 

26.  Interpreting  Information 

.67 

.21 

.58 

0.89 

.20 

.83 

-4.01* 

27.  Communicating,  Internal 

.90 

.15 

.64 

1.00 

.00 

n/a 

-3.53* 

28.  Communicating,  External 

.73 

.23 

.81 

0.90 

.21 

.84 

-2.89* 

29.  Establishing  Relationships 

.92 

.16 

.75 

0.99 

.06 

.65 

-2.17* 

30.  Assisting  Others 

.81 

.16 

.33 

0.83 

.18 

.62 

-0.44 

31 . Selling  or  Influencing 

.58 

.22 

.60 

0.71 

.23 

.66 

-2.16* 

32.  Resolving  Conflicts 

.69 

.23 

.77 

0.83 

.24 

.81 

-2.23* 

33.  Working  with  the  Public 

.59 

.27 

.70 

0.73 

.29 

.84 

-1.87 

34.  Coordinating  Others’  Work 

.65 

.22 

.73 

0.87 

.16 

.45 

-4.28* 

35.  Developing  Teams 

.66 

.20 

.70 

0.60 

.27 

.68 

0.94 

36.  Teaching  Others 

.72 

.19 

.54 

0.76 

.20 

.55 

-0.77 

37.  Directing  Subordinates 

.53 

.24 

.81 

0.49 

.33 

.83 

0.52 

38.  Developing  Others 

.72 

.18 

.55 

0.73 

.22 

.60 

-0.19 

Table  6-15  (continued) 

Comparison  Between  Incumbent  and  Analyst  Descriptive  Statistics  Across  All  Occupations  for 
the  Level  Scale  Rescored  Dichotomouslv:  Generalized  Work  Activities 


Incumbent Analyst 


Descriptor 

M 

SD 

Ik 

M . 

SD 

Ik 

t 

39.  Providing  Consultation 

.63 

.20 

.64 

0.83 

.23 

.78 

-3.47* 

40.  Performing  Admin.  Tasks 

.66 

.21 

.75 

0.93 

.12 

.63 

-5.90* 

41.  Staffing  Org.  Units 

.33 

.25 

:85 

0.30 

.35 

.92 

0.37 

42.  Monitoring  Resources 

.39 

.25 

.79 

0.89 

.14 

.45 

-9.23* 

Note.  Incumbent  statistics  are  based  on  35  occupations  with  Generalized  Work  Activities 
questionnaire  responses  from  at  least  four  incumbents  (mean  number  of  incumbents  =18.91, 
median  = 13,  harmonic  mean  = 9.68).  Analyst  statistics  are  based  on  the  same  35  occupations 
with  Generalized  Work  Activities  questionnaire  responses  from  at  least  six  analysts  (mean 
number  of  analysts  = 10.1 1,  median  =12,  harmonic  mean  = 8.37). 

The  estimate  of  reliability  was  obtained  by  calculating  the  intraclass  correlation  for  k ratings 
across  occupations:  ICC(l,k)  = IBMS-WMSl/BMS  (Shrout  & Fleiss,  1979),  where  k is  the 
harmonic  mean  of  the  number  of  ratings  provided  on  each  occupation. 

The  t statistic  tests  for  differences  in  the  incumbent  and  analyst  group  means. 

*P  < .05 
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Table  6-16 


x minium  ^uinpuiiciui  /uiaivai*  ran 
Descriptor 

icm  iviamx  ior 

uie  /uiaivst  ue\ 

rzi  dealer  uenerali 
Factor 

Lzed  Work  Activities 

FI 

F2 

F3 

Communality 

1.  Getting  Information 

.94 

.12 

-.14 

.91 

2.  Identifying  Objects 

.88 

.17 

-.15 

.83 

4.  Inspecting  Equipment 

.33 

-.50 

.71 

.86 

5.  Estimating  Characteristics 

.88 

-.04 

.16 

.80 

3.  Monitoring  Processes 

.82 

.05 

.37 

.81 

6.  Judging  the  Qualities 

.92 

.17 

-.04 

.87 

7.  Evaluating  Information 

.85 

.05 

-.27 

.80 

8.  Processing  Information 

.78 

.00 

-.53 

.89 

9.  Analyzing  Data 

.92 

.09 

-.25 

.92 

10.  Making  Decisions 

.92 

.13 

-.22 

.90 

1 1 . Thinking  Creatively 

.86 

.01 

-.02 

.75 

12.  Using  Job  Knowledge 

.92 

-.09 

-.01 

.86 

13.  Developing  Objectives 

.87 

.31 

-.12 

.87 

16.  Performing  Phys.  Work  Tasks 

-.10 

.03 

.92 

.86 

14.  Scheduling  Work 

.79 

.37 

-.22 

.81 

17.  Handling  Objects 

-.27 

-.29 

.77 

.75 

15.  Organizing  and  Planning 

.91 

.24 

-.09 

.90 

18.  Controlling  Machines 

-.18 

-.53 

.58 

.65 

19.  Interacting  with  Computers 

.70 

-.23 

-.51 

.80 

20.  Operating  Vehicles 

-.22 

.06 

.66 

.48 

21.  Specifying  Equipment 

.58 

-.62 

.13 

.74 

22.  Implementing  Ideas 

.94 

.00 

.08 

.89 

23.  Repairing,  Mechanical 

-.06 

-.55 

.72 

.83 

24.  Repairing,  Electronic 

.37 

-.50 

.58 

.72 

25.  Documenting  Information 

.81 

.16 

-.37 

.82 

26.  Interpreting  Information 

.85 

.20 

-.32 

.87 

27.  Communicating,  Internal 

.89 

.19 

-.27 

.91 

28.  Communicating,  External 

.55 

.64 

-.38 

.85 

29.  Establishing  Relationships 

.57 

.64 

-.33 

.85 

31.  Selling  or  Influencing 

.43 

.61 

-.20 

.60 

30.  Assisting  Others 

.17 

.61 

.14 

.42 

32.  Resolving  Conflicts 

.64 

.68 

-.17 

.90 

33.  Working  with  the  Public 

.15 

.79 

-.24 

.71 

34.  Coordinating  Others’  Work 

.87 

.39 

-.09 

.92 

35.  Developing  Teams 

.86 

.40 

-.05 

.89 

36.  Teaching  Others 

.79 

.40 

-.01 

.79 

37.  Directing  Subordinates 

.82 

.40 

-.08 

.83 

39.  Providing  Consultation 

.93 

.19 

-.17 

.93 

40.  Performing  Admin.  Tasks 

.76 

.35 

-.43 

.89 

38.  Developing  Others 

.82 

.45 

.01 

.87 

41.  Staffing  Org.  Units 

.71 

.38 

-.09 

.65 

42.  Monitoring  Resources 

.66 

.46 

-.13 

.66 

Percent  of  Variance 

52 

14 

13 

Eigenvalue 

25.02 

6.14 

2.71 
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Table  6-16  (continued) 

Principal  Components  Analysis  Pattern  Matrix  for  the  Analyst  Level  Scale:  Generalized  Work  Activities 

Note.  N = 35.  The  correlation  matrix  was  based  on  means  calculated  at  the  job  level.  FI  = Working  with 
Information,  F2  = Working  with  and  Directing  the  Activities  of  Others,  and  F3  = Manual  and  Physical 
Activities:  Performing  Repair  and  Other  Physical  Work.  These  loadings  are  based  on  an  orthogonal 
varimax  rotation. 
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Table  6-17 

Correlations  Among  Group  Mean  Ratings  Profiles  and  Within  Three  Example  Occupations:  Generalized 
Work  Activities 


Occ.  1 

Occ.  1 

Occ.  2 

Occ.  2 

Occ.  3 

Occ.  3 

Group  1 

Group  2 

Group  1 

Group  2 

Group  1 

Group  2 

(0  = 30) 

(0  = 29) 

(0=33) 

(n  = 32) 

(0  = 44) 

(0  = 44) 

Occ.  1 Group  1 

1.00 

Occ.  1 Group  2 

.96 

1.00 

Occ.  2 Group  1 

.70 

.71 

1.00 

Occ.  2 Group  2 

.72 

.72 

.96 

1.00 

Occ.  3 Group  1 

.71 

.72 

.96 

.93 

1.00 

Occ.  3 Group  2 

.72 

.73 

. .95 

.94 

.98 

1.00 

Note.  Within  each  occupation,  raters  were  randomly  grouped  into  halves. 

Occ.  1 = First-line  Supervisors  and  Managers/Supervisors,  Clerical  and  Administrative  Support  Workers, 
Occ.  2 = Secretaries  Except  Legal  and  Medical,  Occ.  3 = General  Office  Clerks. 
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The  American  Institutes  for  Research  and  its  subcontractors,  Personnel  Decisions 
Research  Institutes,  Inc.,  Management  Research  Institute,  Inc.,  Jeanneret  & Associates,  Inc.,  and 
Westat,  Inc.,  performed  the  work  described  herein  under  Contract  Number  94-542,  administered 
by  the  Utah  Department  of  Employment  Security,  on  behalf  of  the  U.S.  Department  of  Labor. 

Under  this  contract,  the  American  Institutes  for  Research  and  its  subcontractors 
developed  an  operational  prototype  for  an  occupational  data  collection,  analysis,  and 
dissemination  system— the  Occupational  Information  Network  or  0*NET— to  replace  the 
Department  of  Labor’s  Dictionary  of  Occupational  Titles.  This  report,  0*NET  Final  Technical 
Report,  submitted  by  the  American  Institutes  for  Research  as  a major  deliverable  under  this 
contract,  describes  the  empirical  evidence  provided  by  the  preliminary  data  collection  effort  for 
the  meaningfulness  of  the  prototype  system.  An  earlier  report,  Development  of  Prototype 
Occupational  Information  Network  (0*NET)  Content  Model,  described  the  development  of  the 
model  underlying  the  0*NET  and  the  design  of  the  questionnaires  used  to  collect  the 
occupational  information.  A separate  report,  0*NET:  An  Information  System  for  the  Workplace. 
Designing  an  Electronic  Infrastructure  (Rose,  Hesse,  Silver,  & Dumas,  1996),  describes  the 
development  of  the  electronic  database  and  provides  technical  documentation  for  the  database. 

The  Holland  Occupational  Codes  and  explanatory  text  included  in  Chapter  10  of  this 
document  are  adapted  and  reproduced  by  special  permission  of  the  Publisher,  Psychological 
Assessment  Resources,  Inc.,  Odessa,  FL  33556,  from  the  Dictionary  of  Holland  Occupational 
Codes,  Second  Edition  by  Gary  D.  Gottfredson,  Ph.D.,  and  John  L.  Holland,  Ph.D.,  Copyright, 
1982,  1989.  Further  reproduction  for  any  purpose  or  by  any  means  is  prohibited  without  the  prior 
written  permission  of  the  Publisher. 

Please  note  that  the  analysis  results  tables  in  Chapters  3 through  11  are  numbered 
uniformly  across  chapters.  Because  some  analyses  are  not  appropriate  for  every  domain, 
some  domains  are  missing  certain  table  numbers.  This  is  intentional.  Please  refer  to 
chapter  2 and,  particularly,  Figure  2-17  for  a listing  of  what  tables  should  appear  in  each 
chapter. 
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The  work  environment  is  one  of  the  most  salient  aspects  of  a person's  job,  and  the  study 
of  this  environment,  or  work  context,  is  a vital  component  of  job  analysis.  A full  understanding 
of  how  work  actually  gets  accomplished  requires  consideration  of  the  environment  in  which  the 
work  occurs.  The  need  for  such  study  is  widely  recognized  in  the  literature,  and  there  are  few  job 
analysis  instruments  that  do  not  measure  some  aspect  of  the  context  in  which  work  takes  place 
(Gael,  1988;  Ghorpade,  1988). 

Although  many  job  analysis  instruments  include  assessment  of  some  aspects  of  the 
psychosocial  and  physical  work  environment,  no  single  instrument  captures  the  full  range  of 
work  context  factors  or  provides  a theoretical  framework  for  contextual  occupational 
characteristics.  The  taxonomy  described  herein  attempts  to  organize  these  factors  into  a coherent 
structure  in  order  to  account  more  fully  for  the  variables  in  the  analysis  of  jobs  throughout  the 
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world  of  work.  Many  of  the  variables  in  the  taxonomy  have  been  assessed  by  standardized  job 
analysis  procedures  or  have  been  examined  in  other  research  forums.  Even  variables  we  have 
included  to  reflect  current  technology  (such  as  the  use  of  Electronic  Mail  [Descriptor  #2h]  as  a 
communication  method)  have,  in  some  cases,  been  assessed  elsewhere.  Thus,  the  objective  was 
to  integrate  the  existing  body  of  research  regarding  contextual  variables  into  an  organized 
structure  that  included  meaningful  constructs  differentiating  between  occupations. 

Background 

Researchers  familiar  with  job  analysis  and  the  nature  of  work  have  argued  that 
individuals  must  adapt  to  the  physical  and  social  environment  rather  than  simply  respond  to  them 
(Cunningham,  1988;  Frost,  1972;  Kochhar  & Armstrong,  1988;  Lopez,  1988;  McCormick, 
Jeanneret,  & Mecham,  1969a,  1969b,  1972;  Rohmert,  1988).  This  position  suggests  that  work 
context  may  be  conceptualized  as  a set  of  moderator  variables  affecting  or  altering  worker 
behavior.  This  view  requires  that  the  physical  and  social  contexts  of  work  be  subjected  to 
thorough  examination  since  they  represent  the  pervading  contexts  in  which  the  work  stimulus 
impacts  the  worker  and  in  which  the  worker  responds  (see  Figure  7-1)  (Boese  & Cunningham, 
1975).  All  work  activities  occur  within  structural,  physical,  and  social  contexts  involving 
interactions  and  relationships  with  other  individuals  and  the  work  environment.  These  structural, 
social,  and  physical  characteristics  of  the  work  environment  are  addressed  as  contextual  factors 
within  this  chapter.  The  interactions  with  other  workers  that  occur  during  the  accomplishment  of 
work  have  been  included  as  Generalized  Work  Activities  (GW As)  as  described  in  Chapter  6. 

The  goal  for  the  taxonomic  structure  is  to  provide  a systematic  approach  to  the  study  of 
work  context  variables  that  will  provide  valuable  information  and  help  differentiate  occupations. 
This  work  context  taxonomy  is  similar  to  McGrath's  (1976)  global  division  of  organizational 
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factors  into  tasks,  roles,  and  settings.  Work  context  is  defined  here  as  non-task-related  factors  of 
work  which  affect  intrapersonal,  interpersonal,  or  work  outcomes.  Based  on  previous  job  analysis 
work,  research  literature,  and  earlier  taxonomic  efforts,  work  context  has  been  divided  into  three 
higher-order  dimensions:  (a)  Interpersonal  Relationships,  (b)  Physical  Work  Conditions,  and 
(c)  Structural  Job  Characteristics.  That  is,  in  addition  to  those  variables  mediated  by  the  work 
activities  and  the  organization,  there  are  three  broad  categories  of  variables  that  can  be  said  to 
impinge  on  the  worker  in  the  immediate  work  environment:  people,  physical  conditions,  and  the 
structure  of  the  work.  The  Interpersonal  Relationships  dimension  includes  aspects  of  the  context 
such  as  communications,  role  relationships,  and  responsibility  for  others,  which  comprise  the 
social  environment  in  which  the  work  takes  place.  Physical  Work  Conditions  are  the  actual 
environmental  conditions  in  which  the  work  is  conducted  (e.g.,  temperature,  pollutants),  the 
hazards  associated  with  the  occupation  and  possible  injuries  (e.g.,  exposure  to  electricity  or  high 
places),  and  the  demands  placed  on  the  worker  in  terms  of  body  positioning  or  required  safety 
equipment.  Structural  Job  Characteristics  are  based  on  the  nature  of  the  work  and  its 
accomplishment,  including  the  criticality  of  the  work,  how  routine  the  work  is,  and  the  pace  and 
scheduling  of  work  activities. 

Taxonomy 

After  a review  of  the  research  literature  describing  individual  facets  of  work  context,  the 
three  first-order  dimensions  were  further  articulated  into  subcategories  to  create  a preliminary 
second-order  taxonomy.  Also,  many  job  analysis  instruments  contain  items  which,  though 
perhaps  not  labeled  as  such,  relate  to  constructs  within  the  domain  of  work  context.  An 
examination  of  these  items  and  the  related  literature  allowed  further  refinement  and,  in  some 
cases,  redefinition  of  aspects  of  the  taxonomy.  This  review  of  the  literature  and  job  analysis 
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instruments  was  utilized  both  to  assess  the  specific  individual  factors  that  fell  within  the  domain 
of  the  three  higher-order  dimensions,  as  well  as  to  "validate"  the  higher-order  dimensions  as 
factors  that  are  or  can  be  researched  and  used  to  differentiate  between  occupations. 

Although  all  of  the  second-order  dimensions  are  assessed  by  multiple  questions,  most  of 
the  work  context  item-level  constructs  are  measured  by  a single  item  or  scale.  For  instance. 
Communication  is  a second-order  dimension  under  Interpersonal  Relationships,  and  there  are 
five  items  assessing  Communication.  However,  each  of  these  five  items  measures  a different  type 
or  aspect  of  Communication.  Also,  due  to  the  type  of  information  being  collected,  some  of  the 
Physical  Work  Conditions  constructs  involve  multiple  ratings  (e.g.,  level,  frequency,  etc.),  but 
these  also  are  arguably  different  aspects  of  the  conditions  being  assessed. 

Given  the  large  amount  of  material  that  falls  under  the  domain  of  work  context,  multiple 
items  for  all  constructs  would  require  an  extremely  large  set  of  questions.  Substantially 
increasing  the  length  of  the  survey  was  impractical,  and  it  was  judged  that  additional  items  would 
not  provide  incremental  utility.  Accordingly,  due  to  the  relative  objectivity  of  most  of  the 
constructs  and  the  existing  research  evidence,  multiple  items  for  each  construct  have  not  been 
developed. 

Research  suggests  that  many  of  the  work  context  variables  proposed  can  be  rated  quite 
reliably  with  single-item  scales.  A number  of  the  item-level  constructs  are  similar  to  PAQ  job 
elements,  OAI  work  elements,  or  Dictionary  of  Occupational  Titles  (DOT;  U.S.  Dept,  of  Labor, 
1991)  ratings,  and  single  item  scales  from  these  instruments  produced  acceptable  to  high 
reliabilities.  As  reviewed  by  McPhail  et  al.  (1995),  the  dimensions  of  the  PAQ  which  correspond 
to  the  work  context  dimensions  have  very  high  inter  rater  and  rate-rerate  reliabilities  (.85  to  .95), 
and  the  job  elements  within  these  dimensions  are  assessed  with  single-item  scales  (McCormick, 
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Mecham,  Jeanneret,  1989).  Reliability  estimates  for  OAI  items  with  similar  content  to  work 
context  taxonomy  items  are  moderate  to  high  (.62  to  .93)  and  certainly  within  acceptable 
standards  (Boese  & Cunningham,  1975).  Further,  the  results  of  an  analysis  of  ratings  from  the 
DOT  indicate  that  many  work  context  constructs  can  be  rated  very  reliably  (.63  to  .94)  without 
the  need  of  multiple  items  for  each  construct  (Geyer,  Hice,  Hawk,  Boese,  & Brannon,  1989). 

The  contextual  taxonomy  includes  factors  believed  to  differentiate  between  occupations 
and  to  provide  meaningful  and  useful  information  about  any  specific  occupation.  In  cases  where 
it  was  concluded  that  particular  contextual  characteristics  did  not  differentiate  meaningfully 
between  occupations  for  any  of  a variety  of  reasons  (e.g.,  overlap  with  other  characteristics, 
difficulty  in  operationalization,  inappropriate  level  of  detail,  etc.),  the  variable  was  combined 
with  other  factors  or  removed  from  the  taxonomy.  In  developing  the  taxonomy,  overlap  with 
other  domains  was  limited.  Variables  that  we  determined  would  be  more  appropriately  measured 
elsewhere  were  excluded  from  the  taxonomy.  However,  given  the  broad  range  of  constructs  in 
the  work  context  domain,  it  is  inevitable  that  some  overlap  occurs.  This  overlap  is  particularly 
obvious  with  respect  to  the  Organizational  Context  domain.  The  differentiation  between  the  work 
context  and  Organizational  Context  factors  involves  the  focus  of  the  items.  The  work  context 
items  focus  on  the  effects  of  these  constructs  on  the  worker,  the  occupation,  or  specific  tasks, 
whereas  the  Organizational  Context  items  are  intended  to  examine  the  effects  on  a broader  scale. 
Where  construct  overlap  occurs,  we  believe  there  is  sufficient  distinction  between  the  items  to 
warrant  inclusion  in  their  respective  surveys. 

The  literature  presented  below  indicates  that  the  various  aspects  of  work  context 
recommended  for  inclusion  in  the  0*NET  are  distinct,  yet  often  related,  facets  of  work  on  which 
occupations  may  be  expected  to  differ  meaningfully,  both  intra-  and  interorganizationally,  and 
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which  are  likely  to  have  substantive  impact  on  important  outcome  variables.  Each  level  of  the 
three-tiered  taxonomy  is  discussed  below.  The  majority  of  items  assessed  are  similar  to  items  or 
constructs  which  are  currently  assessed  by  various  job  analysis  instruments.  A matrix  showing 
the  overlap  of  our  items  with  constructs  evaluated  in  frequently  used  job  analysis  instruments  is 
presented  in  Figure  7-2. 

Sample  and  Measures 

As  described  above,  based  on  review  of  the  relevant  literature  and  job  analysis 
instruments,  work  context  was  divided  into  three  first-order  dimensions:  (a)  Interpersonal 
Relationships,  (b)  Physical  Work  Conditions,  and  (c)  Structural  Job  Characteristics.  These 
dimensions  were  further  divided  into  second-order  dimensions  from  which  specific  item-level 
constructs  were  generated  (see  Figure  7-3).  An  outline  of  the  taxonomic  structure,  including  item 
level  constructs,  definitions,  and  measurement  scale  types  is  presented  in  Figure  7-4. 
Interpersonal  Relationships 

Interpersonal  Relationships  describe(s)  the  context  of  the  job  in  terms  of  human 
interaction  processes.  Evans,  Johansson,  and  Carrere  (1994)  discussed  the  psychosocial 
environment  as  the  social  climate  of  the  workplace,  the  settings  produced  by  the  activities  of  the 
organization,  and  the  people  in  those  settings.  This  definition  seems  to  include  the  types  of  social 
relationships  and  roles  the  job  holder  must  assume  as  part  of  the  job,  including  communication 
and  accountability  for  others'  performance.  The  Interpersonal  Relationships  dimension  is  divided 
into  four  second-order  factors:  (a)  Communication,  (b)  Role  Relationships,  (c)  Responsibility  for 
Others,  and  (d)  Conflictual  Contact. 
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Physical  Work  Conditions 

The  actual  physical  conditions  in  which  an  employee  is  asked  to  perform  a job  are 
arguably  the  most  obvious  aspects  of  a taxonomy  of  work  context.  There  are  few,  if  any,  job 
analysis  instruments  which  do  not  consider  the  tangible  aspects  of  the  work  environment. 
Physical  Work  Conditions  are  considered  to  be  the  relationships  or  interactions  between  the 
worker  and  the  physical  job  environment.  Evans  et  al.  (1994)  defined  physical  characteristics  of 
work  as  the  inanimate  facets  of  the  work  environment.  For  the  proposed  taxonomy.  Physical 
Work  Conditions  include  (a)  the  Work  Setting,  (b)  the  Environmental  Conditions  of  the  work 
setting  that  may  pose  a hazard  to  the  worker,  and  (c)  Job  Demands,  including  body  positions  and 
work  attire  that  are  part  of  the  work  environment.  Aspects  of  these  factors  are  measured  by  the 
frequency  with  which  a job  exposes  the  worker  to  various  Work  Settings,  certain  Environmental 
Conditions  and  job  hazards,  as  well  as  the  possibility  and  impact  of  injuries.  The  specific  facets 
of  Physical  Work  Conditions  included  in  the  taxonomy  were  obtained  mainly  through  review  of 
the  human  factors  literature,  as  well  as  an  examination  of  several  job  analysis  instruments. 

Structural  Job  Characteristics 

Facets  of  Structural  Job  Characteristics  have  been  referred  to  as  work  context  factors  in 
the  job  analysis  literature  (Bemis,  Belenky,  & Soder,  1983;  McCormick,  1979)  and  are  assessed 
by  several  job  analysis  instruments  (e.g.,  GWI;  JDS;  PAQ;  PMPQ).  Included  in  this  category  are 
assessments  of  the  extent  to  which  the  tasks  are  critical  to  the  organization  and  whether  the  work 
is  routine  or  varied  in  nature,  as  well  as  descriptions  of  work  hours,  scheduling,  the  pace  of  work, 
and  whether  the  job  creates  competition.  Research  has  shown  that  incumbents'  reports  of  these 
types  of  job  characteristics  correlate  significantly  with  several  outcomes,  such  as  job  satisfaction, 
work  frustrations,  anxiety  on  the  job,  turnover  intentions,  and  number  of  physician  visits  (Spector 
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& Jex,  1991).  The  characteristics  examined  include  frustrating  circumstances,  degree  of 
automation,  responsibility  level,  and  decision  latitude.  These  specific  facets  of  the  Structural  Job 
Characteristics  dimension  were  obtained  both  through  a review  of  job  analysis  instruments  and 
relevant  literature,  as  well  as  by  rationally  identifying  the  aspects  of  job  structure  on  which 
occupations  may  differ. 

Summary 

As  discussed  above,  the  contextual  factors  of  work  are  clearly  important  features  on 
which  occupations  may  differ.  In  conjunction  with  organizational  variables  and  Generalized 
Work  Activities,  contextual  factors  affect  many  important  work  outcomes  (see  Figure  7-5). 
Contextual  factors  have  been  linked  to  such  outcome  variables  as  job  performance,  satisfaction, 
compensation,  group  formation,  group  cohesion,  organizational  effectiveness,  and  physical  and 
psychological  health.  However,  attempts  to  assess  these  factors  have  been  unstructured  and  have 
lacked  a systematic  approach.  Although  many  job  analysis  instruments  tap  various  aspects  of  the 
psychosocial  work  environment,  they  do  not  capture  or  assess  individually  the  broad  range  of 
work  context  factors.  The  proposed  taxonomy  builds  on  research  literature  and  existing  job 
analysis  instruments  to  organize  relevant  work  context  factors  into  a coherent  structure  for  job 
analytic  purposes.  The  assessment  of  these  contextual  factors  is  supported  by  numerous  job 
analysis  studies  and  will  provide  valuable  information  for  a variety  of  human  resource 
management  functions. 

Sample 

As  described  in  the  Methods  chapter,  two  samples  of  raters  were  obtained:  job 
incumbents  and  occupational  analysts.  For  some  occupations,  supervisors  were  included  in  the 
job  incumbent  sample,  and  their  responses  are  included  in  this  report.  Ratings  by  job  incumbents 
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and  supervisors  will  be  referred  to  collectively  as  the  incumbent  sample  and  will  serve  as  the 
primary  focus  of  the  analyses. 

Incumbents  from  8 1 occupations  completed  the  work  context  survey.  The  data  collection 
resulted  in  37  occupations  for  which  at  least  four  incumbents  provided  ratings  on  the  work 
context  survey.  Only  data  from  these  occupations  were  included  in  the  analyses,  yielding  a total 
of  728  incumbent  responses.  A list  of  the  37  occupations  and  the  number  of  respondents  within 
each  is  provided  in  Figure  7-6.  As  noted  in  previous  chapters,  the  occupations  in  this  sample  span 
a wide  range  of  occupations  and  were  sampled  from  many  different  organizations. 

In  addition  to  the  job  incumbent  sample,  ratings  from  six  occupational  analysts  were 
collected.  Experienced  analysts  from  the  North  Carolina  Occupational  Analysis  Field  Center 
rated  selected  portions  of  the  work  context  survey  for  the  occupations  included  in  their  analyses. 
The  analysts  reviewed  commonly  performed  tasks  from  the  DOT  for  each  of  the  occupations, 
then  rated  the  occupations  on  the  work  context  variables.  Although  six  analysts  rated  all  81 
occupations,  only  those  37  occupations  for  which  there  were  at  least  four  incumbent  respondents 
were  used  in  the  data  analyses.  A review  of  the  DOT  job  tasks  without  actually  observing  the  job 
being  performed  did  not  provide  sufficient  information  regarding  several  of  the  work  context 
variables,  and  these  variables  were  not  rated  by  the  analysts.  For  instance,  the  formality  of 
communication,  communication  methods,  and  Privacy  of  Communication  (Descriptor  #5)  are 
difficult  to  evaluate  based  simply  on  a list  of  job  tasks,  and  would  require  observation  before 
accurate  assessments  could  be  performed.  In  addition,  as  noted  in  Chapter  2,  analysts  rated  items 
having  to  do  with  frequency  on  a five-point  (0  to  4)  scale,  while  incumbents  made  frequency 
ratings  on  scales  of  varying  length. 

Descriptive  Statistics 

The  mean,  standard  deviation,  standard  error  of  measurement,  and  interrater  agreement 
coefficient  for  each  variable  are  provided  in  Table  7-1.  Unlike  the  descriptors  in  many  of  the 
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other  domains,  the  work  context  variables  are  not  measured  on  common  rating  scales.  Thus  a 
comparison  of  means  across  all  items  is  not  appropriate. 

A review  of  the  mean  values  for  each  descriptor  indicates  that  the  most  commonly  used 
communication  method  across  the  occupations  was  Face-to-Face  Interactions  with  other 
individuals  (Descriptor  #2a),  followed  closely  by  use  of  a Telephone  (Descriptor  #2f).  The 
communication  methods  with  the  lowest  ratings  were  Video  Conferencing  (Descriptor  #2d)  and 
Public  Speaking  (Descriptor  #2c).  Even  with  the  innovations  in  computer  technology,  the  use  of 
Electronic  Mail  (Descriptor  #2h)  and  Interactive  Computer  (Descriptor  #2g)  communication  was 
rated  relatively  low  for  the  occupations  in  this  sample.  In  general,  communication  on  the  job  was 
viewed  as  more  subjective  than  objective  in  nature. 

Across  all  occupations  in  the  sample,  the  types  of  job  interaction  rated  most  important 
were  Working  With  or  Contributing  To  a Team  (Descriptor  #6e)  and  Providing  a Service  to 
Others  (Descriptor  #6c).  These  ratings  reflect  the  fact  that  these  types  of  interactions  can  be 
found  in  a wide  range  of  occupations,  ranging  from  janitors  to  managers.  The  two  types  of  Job 
Interactions  (Descriptor  #6)  rated  least  important  were  Persuading  and  Influencing  Others 
(Descriptor  #6b)  and  Taking  a Position  Opposed  to  Others  (Descriptor  #6d).  This  may  reflect  the 
types  of  occupations  included  in  this  sample.  Of  the  37  occupations,  relatively  few  explicitly 
involve  selling  or  influencing  others. 

The  type  of  Work  Setting  (Descriptor  #12)  in  which  workers  from  this  sample  spent  the 
most  time  was  Indoors  (Descriptor  #12a)  in  a climate  controlled  environment.  The  Physical 
Proximity  (Descriptor  #14)  among  workers  is  somewhat  close,  and  Environmental  Conditions 
(Descriptor  #15)  was  the  highest  rated  environmental  condition.  Making  Repetitive  Motions 
(Descriptor  #22),  Standing  (Descriptor  #22b).  Sitting  (Descriptor  #22a),  and  Handling  Tools  and 
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Objects  (Descriptor  #22g)  were  the  most  frequently  occurring  Body  Positioning  (Descriptor  #22) 
activities  for  workers  in  this  sample.  Exposure  to  various  job  hazards  received  relatively  low 
ratings,  with  Exposure  to  Hazardous  Situations  (Descriptor  #2 la)  (those  likely  to  involve  cuts, 
bites,  or  minor  bums)  receiving  the  highest  mean  rating  of  the  group. 

The  general  findings  regarding  the  means  and  standard  deviations  reflect  what  is 
commonly  known  about  how  and  where  work  is  performed.  A majority  of  jobs  are  performed 
indoors,  with  relatively  few  associated  hazards.  Individual  communication,  either  Face-to-Face 
(Descriptor  #2a)  or  via  Telephone  (Descriptor  #2f),  remains  the  most  frequently  used  technique 
to  convey  information.  One  finding  which  may  reflect  somewhat  recent  changes  in  the  way  work 
is  structured  is  the  high  mean  rating  for  working  with  and  contributing  to  a team  (Work/ 
Contribute  to  Team,  Descriptor  #6e).  Employees  at  all  levels  of  organizations  find  it  relatively 
important  to  participate  on  teams  in  order  to  perform  work  activities. 

Reliability 

As  reported  in  Table  7-1,  based  on  a harmonic  mean  of  9.44  respondents  per  occupation, 
most  variables  had  an  acceptable  level  of  interrater  agreement.  The  median  interrater  agreement 
coefficient  was  .83,  with  a range  from  .20  to  .97.  Given  the  relatively  small  number  of  raters  per 
occupation,  the  agreement  coefficients  are  acceptable  for  most  items.  Only  five  items  had  an 
agreement  coefficient  less  than  .50  and  another  nine  items  had  reliabilities  above  .50,  but  less 
than  .70.  However,  a somewhat  large  number  of  raters  is  desired  when  evaluating  the  agreement 
of  descriptors.  Table  7-2  provides  estimates  of  single-rater  reliability  and  reliability  which  would 
have  been  obtained  if  the  initial  sampling  plan  of  30  raters  per  occupation  was  obtained. 
Obviously,  the  single-rater  estimates  were  lower  than  those  found  using  either  the  harmonic 
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mean  number  of  raters  or  the  estimate  found  for  30  raters.  The  estimates  for  30  raters  provide 
extremely  high  levels  of  reliability  (.44  to  .99). 

When  comparing  these  results  to  the  median  reliabilities  reported  for  work  context 
variables  obtained  using  other  procedures,  we  find  the  present  results  to  be  very  satisfactory. 
Specifically,  the  median  reliability  for  PAQ  dimensions  that  reflect  the  work  context  range  from 
.86  to  .99  (n  = 19,961  analyzed  pairs)  (McCormick  et  al.,  1989).  Reliability  estimates  based  on 
use  of  the  OAI  to  analyze  work  context  range  from  .62  to  .93  (Boese  & Cunningham,  1975),  and 
ratings  of  DOT  work  context  items  by  four  trained  analysts  range  from  .63  to  .94  (Geyer  et  al., 
1989). 

Responses  to  items  regarding  Work  Setting  (Descriptor  #12),  Work  Attire  (Descriptor 
#23),  Body  Positioning  (Descriptor  #22),  Environmental  Conditions  (Descriptor  #15),  and  job 
hazards  resulted  in  the  most  consistently  high  levels  of  agreement.  Overall,  the  level  of 
agreement  found  for  the  work  context  variables  is  adequate  for  research  purposes,  and  only  a few 
items  have  reliability  coefficients  below  .70.  The  agreement  coefficients  based  on  the  average 
raters  per  occupation  showed  acceptable  agreement,  and  it  would  be  expected  that  samples  of  this 
size,  and  certainly  samples  as  large  as  30  judges,  would  provide  satisfactory  levels  of  reliability. 

The  lower  agreement  coefficients  for  some  descriptors  could  be  due  to  a number  of 
causes.  It  is  possible  that  the  wording  of  the  item  was  ambiguous,  or  that  job  incumbents  are 
unable  to  reliably  provide  information  regarding  certain  variables.  The  descriptor  with  the  lowest 
agreement  estimate  is  the  use  of  video  conferencing  as  a communication  method  (Video 
Conference,  Descriptor  #2d).  The  infrequent  use  of  this  method  may  account  for  the  low  level  of 
agreement.  It  is  also  possible  that  some  incumbents  did  not  understand  the  term  Video 
Conferencing  (Descriptor  #2d).  Janitors  rated  the  item  almost  as  high  as  did  executives.  It  is 
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likely  that  some  incumbents  interpreted  the  question  to  include  videotaped  addresses  by  mangers 
and  executives.  Further  review  of  those  items  with  questionable  interrater  reliability  coefficients 
is  certainly  warranted  and  would  be  completed  for  a revised  work  context  survey. 

Unlike  the  other  domains  in  the  content  model,  all  work  context  variables  are  not 
assessed  on  a common  rating  scale.  Some  items  are  rated  on  a level  scale  where  a not  relevant 
response  may  be  appropriate.  However,  several  items  involve  topics  which  do  not  lend 
themselves  to  a not  relevant  rating.  For  instance,  the  item  concerning  Physical  Proximity 
(Descriptor  #14)  to  coworkers  cannot  have  a not  relevant  response.  The  use  of  a not  relevant 
response  has  been  thought  to  increase  the  level  of  interrater  agreement;  thus  it  is  of  interest  to 
examine  the  impact  of  the  not  relevant  response  on  interrater  agreement  in  the  current  sample. 
For  those  items  where  a not  relevant  response  was  available,  two  additional  interrater  agreement 
coefficients  were  calculated.  One  estimate  was  based  on  a relevant/not  relevant  dichotomy  and 
the  other  based  only  on  relevant  responses.  These  interrater  agreement  estimates  are  consistent 
with  and  largely  similar  to  those  found  using  the  full  scale  scoring.  Both  the  dichotomously 
coded  responses  and  the  relevant-only  responses  resulted  in  slightly  lower  agreement  coefficients 
than  the  full  scale  coding.  Due  to  the  small  number  of  items  with  not  relevant  options  and  the 
similarity  of  these  results  to  the  findings  using  the  full-scale  responses,  no  table  was  created  to 
present  these  results. 

Analyses  of  Variance 

As  another  means  of  assessing  the  level  of  interrater  agreement,  an  analysis  of  variance 
was  completed  to  examine  the  descriptors,  occupations,  and  interactions  as  sources  of  variance 
treating  the  descriptors  as  repeated  measures  variables.  Significant  differences  were  found  for  the 
descriptors  across  occupations.  Occupations  and  the  descriptors  by  occupation  interaction  also 


7-14  Strong,  Jeanneret,  McPhail,  & Blakley 


yielded  significant  effects.  These  results  indicate  that  some  variables  were  very  useful  in 
differentiating  between  occupations.  We  note  that  most  of  the  90  work  context  descriptors  are 
rated  on  seven-  or  eight-point  scales,  but  about  20  descriptors  are  rated  on  four-  or  five-point 
scales.  These  differences  in  scale  length  will  operate  to  produce  between-descriptor  variance  that 
is  artifactual.  However,  such  differences  would  have  no  impact  on  the  most  salient  source  of 
variance  in  Table  7-4— descriptors  by  occupations.  Interrater  agreement  coefficients  were 
obtained  based  on  these  sources  of  variation.  As  reported  in  Table  7-5,  the  full  sample  interrater 
agreement  is  high,  as  is  the  estimate  for  a sample  of  30  raters  per  occupation. 

Unlike  the  variables  in  other  domains,  the  work  context  variables  do  not  lend  themselves 
to  aggregation  at  higher  level  factors.  The  nature  of  the  constructs  assessed  and  the  use  of 
different  rating  scales  makes  it  inappropriate  to  attempt  to  formulate  higher-order  scale  scores. 
Thus,  analyses  reported  in  other  chapters  were  not  appropriate  for  this  chapter. 

In  general,  the  results  of  the  various  interrater  agreement  analyses  lead  to  the  conclusion 
that  the  work  context  descriptors  can  be  assessed  with  sufficient  levels  of  agreement  by  job 
incumbents.  As  always,  a large  sample  of  respondents  per  occupation  is  recommended.  There 
was  an  average  of  approximately  20  incumbents  per  occupation  in  the  current  sample  (harmonic 
mean  = 9.44),  and  it  appears  that  a sample  of  that  size,  and  certainly  a sample  of  30  raters  per 
occupation  would  result  in  adequate  interrater  agreement. 

Descriptor  Relationships 

The  variables  in  the  other  domain  chapters  were  assessed  using  multiple  scales  (e.g., 
importance,  level,  job  entry  requirements).  Not  all  of  the  work  context  variables  lend  themselves 
to  measurement  by  such  scales,  and  an  analysis  of  the  relationship  between  scale  types  is  not 


feasible  for  these  variables. 
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The  intercorrelations  between  the  work  context  variables  at  the  occupation  level  are 
presented  in  Tables  7-9a,  7 -9b,  and  7-9c.  The  intercorrelations  in  the  three  tables  are  computed 
within  each  of  the  three  higher-order  work  context  dimensions  (Interpersonal  Relationships, 
Physical  Work  Conditions,  and  Structural  Job  Characteristics)  and  not  across  dimensions.  Tables 
7-10a,  7-10b,  and  7-10c  present  the  same  intercorrelations  at  the  individual  respondent  level 
using  four  randomly  selected  raters  per  occupation.  The  primary  focus  of  this  study  is  at  the 
occupation  level,  and  the  individual  level  data  are  presented  for  comparison  purposes,  but  are  not 
discussed  in  detail. 

The  occupation  level  intercorrelations  within  the  Interpersonal  Relationship  dimension 
yield  meaningful  patterns  of  relationships.  For  instance,  responsibility  for  others’  work 
(Responsible  Others’  Work,  Descriptor  #8)  is  strongly  related  to  the  occupation  interaction  of 
Supervising  and  Developing  Others  (Descriptor  #6a)  (r  = .66)  and  to  Coordinating  and  Leading 
Others  (Descriptor  #6g)  (r  = .61).  Dealing  With  the  Public  (Descriptor  #6f)  is  strongly  related  to 
Social  Interaction  (Descriptor  #4)  (r  = .60),  Persuading  and  Influencing  Others  (Descriptor  #6b) 

(r  = .66),  Public  Speaking  (Descriptor  #2c)  (r  = .57),  and  interacting  with  Unpleasant  Individuals 
(Descriptor  #10)  (r  = .53),  but  weakly  related  to  Communication  Privacy  (Descriptor  #5)  (r  = - 
.04)  and  Communication  Formality  (Descriptor  #1)  (r  = .05). 

Intercorrelations  within  the  Physical  Work  Conditions  dimension  also  display  rational 
patterns  of  relationships.  Work  Setting:  Indoors,  Controlled  (Descriptor  #12a)  has  a strong 
negative  correlation  with  descriptors  measuring  uncomfortable  working  conditions  (r  = -.47  to  r 
= -.79)  and  has  a positive  relationship  with  Business  or  Office  Attire  (Descriptor  #23a)  (r  = .47). 
Privacy  of  Work  Area  (Descriptor  #13)  was  positively  related  to  the  Sitting  (Descriptor  #22a) 
body  position  (r  = .49),  but  negatively  related  to  all  other  Body  Positioning  (Descriptor  #22)  (r  = 
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-.23  to  r -.71).  Sitting  (Descriptor  #22a)  is  often  associated  with  an  office  job,  which  would 
likely  provide  more  privacy  than  jobs  requiring  Walking  (Descriptor  #22d).  Standing  (Descriptor 
#22b),  or  Handling  Tools  (Descriptor  #22g).  Exposure  to  Diseases  and  Infections  (Descriptor 
#17a)  is  positively  related  to  wearing  a Special  Uniform  (Descriptor  #23b)  (r  = .72)  and  Physical 
Proximity  (Descriptor  #14)  (r  = .56),  which  may  reflect  duties  found  in  nursing  and  public  safety 
jobs. 

Rational  patterns  of  intercorrelations  are  also  found  within  the  Structural  Job 
Characteristics  dimension.  Machine  Driven  Pace  (Descriptor  #39)  is  positively  correlated  with 
Repetitive  Activities  (Descriptor  #34)  (r  = .59)  and  working  under  Time  Pressure  or  Deadlines 
(Descriptor  #37)  (r  = .66),  but  negatively  correlated  with  Unstructured  Tasks  (Descriptor  #35)  (r 
= -.63).  Level  of  Automation  (Descriptor  #29)  on  the  job  is  positively  related  to  the  necessity  of 
being  Accurate  and  Exact  (Descriptor  #3 1)  (r  = .55)  and  Attention  to  Details  (Descriptor  #32)  (r 
— .55).  Dealing  with  Frustrating  Circumstances  (Descriptor  #28)  is  negatively  related  to  Clarity 
of  Tasks  or  Performance  goals  (Descriptor  #30)  (r  = -.37)  and  positively  related  to  Accountability 
for  Results  (Descriptor  #26)  (r  = .70)  and  Decision  Latitude  (Descriptor  #27)  (r  = .77). 

The  number  of  variables  within  the  work  context  domain  does  not  allow  for  presentation 
of  correlations  of  variables  across  the  higher-order  dimensions.  However,  there  are  relationships 
worth  noting.  For  instance,  the  importance  of  Supervising  Others  (Descriptor  #6a)  is  significantly 
correlated  with  Consequence  of  Error  (Descriptor  #24)  (r  = .42),  Impact  (Descriptor  #25)  (r  = 

.52),  Frequency  (Descriptor  #25b)  (r  = .33),  and  Latitude  of  Decisions  (Descriptor  #27)  (r  = .37), 
and  Accountability  of  Results  (Descriptor  #26)  (r  = .42).  Further,  Supervising  Others  (Descriptor 
#6a)  was  significantly  related  to  the  extent  that  work  is  Unstructured  (Descriptor  #35)  (r  = .36). 
Formality  of  Communication  (Descriptor  #1)  is  related  to  Consequence  of  Error  (Descriptor  #24) 
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(r  = .34),  Impact  of  Decisions  (Descriptor  #25)  (r  = .44)  and  Frequency  of  Decisions  (Descriptor 
#25b)  (r  = .34).  The  importance  of  dealing  with  Physically  Aggressive  (Descriptor  #11) 
individuals  is  also  correlated  with  Consequence  of  Error  (Descriptor  #24)  (r  = .33)  and  Impact  of 
Decisions  (Descriptor  #25)  (r  = .36). 

Responsibility  for  the  Health  and  Safety  of  Others  (Descriptor  #7)  is  significantly 
correlated  with  Exposure  to  Diseases  and  Infections  (Descriptor  #17a)  (r  = .39),  High  Places 
(Descriptor  #18a)  (r  = .35),  Hazardous  Conditions  (Descriptor  #19a)  (r  = .41),  Hazardous 
Equipment  (Descriptor  #20a)  (r  = . 49),  and  Hazardous  Situations  (Descriptor  #2 la)  (r  = .67). 
Responsibility  for  Health  and  Safety  (Descriptor  #7)  is  also  correlated  with  wearing  a Special 
Uniform  (Descriptor  #23b)  (r  = .42)  and  Common  Safety  Attire  (Descriptor  #23d)  (r  = .58). 
Importance  of  being  Constantly  Aware  (Descriptor  #33)  of  events  is  correlated  with  several 
Environmental  Conditions  (Descriptor  #15),  such  as  Extreme  Temperature  (Descriptor  #15b)  (r  = 
.33),  Poor  Lighting  (Descriptor  #15c)  (r  = .55),  Contaminants  (Descriptor  #15d)  (r  = .39),  and 
Exposure  to  Diseases  and  Infections  (Descriptor  #17a)  (r  = .41).  The  extent  to  which  the  work 
pace  is  machine  driven  is  correlated  with  Exposure  to  Hazardous  Equipment  (Descriptor  #20a)  (r 
= .43)  and  Hazardous  Conditions  (Descriptor  #19a)  (r  = .54). 

Factor  Structure 

After  finding  rational  patterns  of  intercorrelations  between  variables,  a principal 
components  factor  analysis  with  varimax  rotation  was  conducted  to  further  explore  the 
relationships  between  items.  The  results  of  this  analysis  (reported  in  Table  7-11)  yielded  a seven- 
factor  solution  for  the  structure  of  the  work  context  variables. 

The  first  factor  is  labeled  environmental  conditions.  Items  associated  with  Work  Setting 
(Descriptor  #12)  and  Environmental  Conditions  (Descriptor  #15)  produced  strong  loadings  on 
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this  factor.  For  example.  Poor  Lighting  (Descriptor  #15c)  (r  = .91),  Extreme  Temperatures 
(Descriptor  #15b)  (r  = .91),  and  working  indoors  (r  = -.79),  Work  Setting:  Indoors,  Controlled 
(Descriptor  12a)  had  very  high  loadings  on  the  environmental  conditions  factor,  as  did  working 
in  an  Open  Vehicle  (Descriptor  #2e)  (r  = .87)  and  Outdoors  (Descriptor  #12c)  under  exposed 
conditions  (r  = .86). 

The  second  factor  is  labeled  physical  activity  and  manual  work  which  consists  mostly  of 
Body  Positioning  (Descriptor  #22)  variables.  For  instance,  Standing  (Descriptor  #22b)  (r  = .93), 
Sitting  (Descriptor  #22a)  (r  = -.93),  and  Handling  Tools  or  Objects  (Descriptor  #22g)  (r  = .50)  all 
have  strong  loadings  on  this  factor.  Also,  the  use  of  Voice  Mail  (Descriptor  #2e)  (r  = -.64)  and 
Electronic  Mail  (Descriptor  #2h)  (r  = -.56)  loaded  negatively  on  this  factor.  The  pattern  of 
positive  and  negative  loadings  on  this  factor  are  consistent  with  aspects  of  manual  labor  and 
physical  tasks  versus  those  work  activities  that  do  not  require  extensive  physical  effort. 

Managerial  and  interpersonal  relations  is  the  third  factor  extracted  from  the  analysis. 
Variables  with  strong  loadings  on  this  factor  include  Persuading  and  Influencing  Others 
(Descriptor  #6b)  (r  = .76),  Coordinating  and  Leading  Activities  (Descriptor  #6g)  (r  = .71), 
Supervising  and  Developing  Others  (Descriptor  #6a)  (r  = .68),  and  Decision  Latitude  (Descriptor 
#27)  (r  = .67).  The  variables  within  this  factor  are  common  to  supervisory  or  management 
positions  or  relating  to  others  on  an  individual  basis. 

The  fourth  factor  is  labeled  structured  and  machine  operations  This  factor  is 
characterized  by  variables  associated  with  performing  automated  or  strictly  defined  tasks.  For 
example,  performing  Repetitive  Activities  (Descriptor  #34)  (r  = .80),  Level  of  Automation 
(Descriptor  #29)  (r  = .72),  Unstructured  Tasks  and  Goals  (Descriptor  #35)  (r  = -.70),  and 
Machine  Driven  Pace  (Descriptor  #39)  (r  = .89)  have  strong  loadings  on  this  factor. 
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The  fifth  factor  extracted  from  the  analyses  includes  variables  found  in  business  or  office 
environments.  Wearing  Business  Clothes  (Descriptor  #23a)  (r  = .61)  and  Communicating  via  the 
Telephone  (Descriptor  #2f)  (r  = .59),  with  Letters  and  Memos  (Descriptor  #2j)  (r  = .58),  with 
Handwritten  Notes  (Descriptor  #2i)  (r  = .56),  and  in  Formal  Communication  (Descriptor  #')  (r  = 
.59)  produced  positive  loadings  on  this  factor.  Wearing  Safety  Equipment  (Descriptor  #23e)  (r  = 
-.55)  and  Maintenance  Clothing  (Descriptor  #23c)  (r  = -.58)  yielded  negative  loadings. 

The  sixth  factor,  health  and  safety  conditions,  is  defined  by  variables  that  can  be 
associated  with  tasks  found  in  health  care  and  public  safety  jobs.  For  instance.  Exposure  to 
Diseases  (Descriptor  #17a)  (r  = .81),  Wearing  a Special  Uniform  (Descriptor  #23b)  (r  = .67),  and 
being  responsible  for  the  Health  and  Safety  of  Others  (Descriptor  #7)  (r  = .50)  produced  strong 
loadings  on  this  factor. 

The  final  factor  extracted  from  the  analysis  is  labeled  interacting  with  the  public. 
Obviously,  Dealing  With  the  Public  (Descriptor  #6f)  (r  = .81),  Public  Speaking  (Descriptor  #2c) 
(r  = .71),  and  Social  Interaction  (Descriptor  #4)  (r  = .55)  were  strongly  loaded  on  this  factor. 
Apparently,  encountering  Unpleasant  Individuals  (Descriptor  #10)  (r  = .53)  also  is  associated 
with  public  interactions.  Interestingly,  Exposure  to  Radiation  (Descriptor  #16a)  has  a loading  on 
this  factor.  The  correlation  was  negative  (r  = -.45),  which  is  logical  given  that  employees 
working  near  Radiation  (Descriptor  #16)  are  unlikely  to  also  work  with  the  public. 

Overall,  the  variable  loadings  can  be  interpreted  as  meaningful  work  context  factors.  We 
also  attempted  to  confirm  our  original  three-dimension  taxonomy  by  limiting  the  principal 
components  analysis  to  the  extraction  of  three  factors.  The  results  are  surprisingly  similar  to  our 
initial  taxonomy  given  our  very  small  sample  size  and  restricted  set  of  occupations. 
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The  factor  analysis  matrix  of  coefficients  is  reported  in  Table  7-1  lb.  The  first  factor  is  the 
physical  work  conditions  dimension  from  the  original  taxonomy,  and  only  two  extraneous  items 
load  on  this  factor  that  are  not  readily  explained  (Deadline  and  Time  Pressure  [Descriptor  #37] 
and  Level  of  Competition  [Descriptor  #36]).  The  second  factor  is  the  original  interpersonal 
relationships  dimension.  This  factor  is  also  extremely  clear  and  has  only  one  extraneous  item, 
namely  that  of  Unstructured  Tasks  and  Goals  (Descriptor  #35).  The  final  component  is  matched 
with  the  structural  job  characteristics  dimension  from  the  initial  taxonomy.  This  is  the  most 
confounded  component  in  that  it  includes  all  of  the  physical  body  positions  (e.g..  Sitting 
[(Descriptor  #22a)].  Standing  [(Descriptor  #22b)])  having  their  expected  positive  or  negative 
signs,  and  interpersonal  items  associated  with  using  computers,  writing  reports,  and  providing 
service.  It  is  interesting  to  speculate  that  all  of  these  activities,  especially  the  use  of  a computer, 
add  more  structure  to  a job  as  opposed  to  facilitating  interpersonal  relationships.  Perhaps  this 

confirmatory  analysis  is  more  accurate  in  describing  the  structure  of  work  context  than  is  our 
original  taxonomy. 

Occupation  Differences 

In  an  attempt  to  further  investigate  how  well  the  variables  differentiate  between 
occupations,  the  mean  item  ratings  from  six  occupations  were  examined.  The  six  occupations 
were  selected  to  represent  distinct  and  diverse  jobs  in  the  overall  economy  and  included  General 
Managers  and  Top  Executives,  Computer  Programmers,  Registered  Nurses,  Police  Patrol 
Officers,  Janitors  and  Cleaners,  and  Maintenance  Repairers  and  General  Utility  Workers.  The 
mean  ratings  of  the  variables  for  each  of  the  six  occupations  are  presented  in  Table  7-12. 

An  examination  of  the  patterns  of  mean  ratings  reveals  meaningful  differences  between 
the  occupations.  For  instance.  Computer  Programmers  produced  the  lowest  level  of  Social 
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Interaction  (Descriptor  #4)  (M  - 3.76,  SD  = 1.22)  and  the  highest  level  of  Making  Repetitive 
Motions  (Descriptor  #22i)  (M  = 2.89,  SD  = 2.09).  Managers  and  Top  Executives  had  the  highest 
levels  of  Supervising  and  Developing  Others  (Descriptor  #6a)  (M  = 4.22,  SD  = .71), 
Accountability  for  Results  (Descriptor  #26)  (M  = 6.13,  SD  = .96),  and  wearing  Business  or 
Office  Clothing  (Descriptor  #23)  (M  = 4.18,  SD  = 1.39).  The  two  occupations  that  most  directly 
involve  decisions  affecting  the  Health  and  Safety  of  Others  (Descriptor  #7),  Registered  Nurses 
and  Police  Patrol  Officers,  yielded  the  highest  levels  of  Consequence  of  Error  (Descriptor  #24) 
(M  = 5.79,  SD  = 1.69  and  M = 5.48,  SD  = 1.90)  and  Impact  of  Decisions  (Descriptor  #25)  (M  = 
5.20,  SD  = 1.42  and  M = 6. 12,  SD  = 1 .51).  Police  Patrol  Officers  also  were  most  frequently 
Interacting  with  the  Public  (Descriptor  #6f)  (M  = 4.80,  SD  = .41)  and  handling  Physical 
Aggression  (Descriptor  #1 1)  (M  = 2.80,  SD  = .76). 

Teamwork  was  reasonably  important  to  all  six  occupations,  but  had  greater  variability 
among  the  Janitor  and  Maintenance  Repairer  Categories.  Communicating  Face-to-Face  With 
Other  Individuals  (Descriptor  #2a)  was  the  most  common  communication  method  across  all  six 
occupations.  Computer  Programmers  and  Janitors  consistently  used  all  communication  methods 
less  frequently  than  did  the  other  occupations.  However,  Computer  Programming  is  apparently 
the  only  occupation  from  this  group  of  six  using  Electronic  Mail  (Descriptor  #2h)  (M  = 4.67,  SD 
= 1.50)  to  any  significant  degree.  Not  surprisingly.  Police  Patrol  Officers,  Janitors  and  Cleaners, 
and  Maintenance  Repairers  work  more  frequently  under  unpleasant  Environmental  Conditions 
(Descriptor  #15)  (e.g..  Noise  [Descriptor  #15a].  Extreme  Temperatures  [Descriptor  #15b])  than 
do  workers  in  the  other  occupations,  and  they  were  less  likely  to  work  in  an  Indoor  Temperature- 
Controlled  area  (Descriptor  #12a). 
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Discriminant  Analyses 

In  order  to  assess  how  well  the  work  context  variables  differentiate  between  occupations, 
a discriminant  analysis  was  conducted  using  the  ratings  provided  by  the  incumbents  in  the  37 
occupations.  The  analysis  yielded  six  interpretable  functions,  and  the  results  are  presented  in 
Table  7-13.  The  results  indicate  that  the  work  context  variables  are  capable  of  differentiating  one 
occupation  from  another.  The  variances  accounted  for  by  the  descriptors,  as  assessed  with  rj2, 
range  from  .06  to  .61.  The  squared  function  coefficients  range  from  .02  to  .50.  It  appears  that 
those  items  involving  Environmental  Conditions  (Descriptor  #15)  and  physical  activities  (i.e., 
working  outdoors,  working  in  a vehicle.  Sitting  [Descriptor  #22a],  and  Standing  [Descriptor 
#22b])  are  more  useful  in  differentiating  occupations,  at  least  for  this  limited  data  set. 

The  variable  loadings  on  the  discriminant  functions  provided  evidence  about  the  structure 
of  the  items  that  differentiate  occupations.  The  six  functions  involve  environmental  conditions 
physical  activity,  health  care,  interacting  with  the  public,  managerial  and  interpersonal  relations. 

hazardous  work  conditions.  Using  these  six  functions,  58%  of  incumbents  were  correctly 
reclassified  into  their  occupations.  Using  all  23  functions,  not  just  the  six  shown  in  Table  7-13, 
79%  of  incumbents  were  correctly  reclassified.  The  six  functions  shown  in  Table  7-13  are  similar 

to  the  factors  as  extracted  by  the  principal  components  factor  analysis  previously  presented  in 
Table  7-1  la. 

Convergence  with  Analysts’  Ratings 

Table  7-14  provides  the  mean  ratings  from  both  the  incumbent  and  analyst  samples.  As 
stated  earlier,  analysts  made  ratings  based  on  a reading  of  tasks  from  the  DOT  for  each  of  the 
occupations  in  the  sample.  Many  work  context  variables  cannot  be  accurately  assessed  from  this 
information,  and  thus  Table  7-14  only  includes  data  for  those  variables  which  analysts  were 
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capable  of  rating.  Also  presented  in  the  table  are  t and  F tests  comparing  the  analyst  and 
incumbent  ratings,  as  well  as  the  correlation  coefficient  and  cf;  index.  Analysts  utilized  a 
modified  frequency  scale  for  some  items,  thus  comparisons  between  incumbent  and  analyst 
frequency  ratings  are  not  reported  for  those  items. 

Analysts’  and  incumbents’  ratings  have  a moderate  level  of  agreement,  with  the  median 
correlation  between  the  incumbent  and  analyst  mean  ratings  being  .58.  Further,  there  were 
significant  differences  in  the  mean  ratings  provided  by  the  two  samples.  Of  the  33  items  with 
comparable  scales,  24  were  rated  significantly  higher  by  incumbents.  Given  the  nature  of  the 
work  context  items,  this  may  not  be  surprising.  The  physical  and  structural  components  of  an 
occupation  are  much  more  salient  or  apparent  to  incumbents  than  they  are  likely  to  be  for 
analysts  reading  a list  of  tasks. 

On  the  other  hand,  the  analyst  data  tended  to  be  just  slightly  more  reliable.  The  median 
reliability  coefficient  for  analysts  is  .86  while  it  is  .83  for  incumbent  ratings.  Although  the 
median  reliabilities  are  not  substantially  different,  the  analysts  have  fewer  items  with 
significantly  low  reliabilities. 

In  an  additional  attempt  to  compare  ratings  from  analysts  and  incumbents,  a principal 
components  factor  analysis  with  varimax  rotation  was  conducted  on  the  analysts’  data.  The 
resulting  factor  structure  (reported  in  Table  7-16)  is  very  similar  to  that  found  for  the  incumbent 
data.  Although  the  analysts  rated  only  a subset  of  the  work  context  items,  these  data  , like  that  for 
the  incumbents,  yielded  a seven-factor  solution  for  the  structure  of  the  variables.  The  factors 
from  the  analyst  data:  (a)  managerial  relations,  (b)  environmental  conditions,  (c)  health  and 
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safety  conditions,  (d)  interacting  with  the  public,  (e)  physical  activity,  (f)  body  movement,  and 
(g)  sjructured/machine  operations,  are  very  similar  to  the  seven  factors  found  with  the  incumbent 
data  presented  in  Table  7-1  la.  Generally,  work  context  items  load  on  the  same  factors  within 
each  data  set.  The  only  substantive  difference  between  the  two  data  sets  is  that  the  incumbent 
data  yielded  a factor  related  to  a business/office  environment,  while  the  analyst  data  yielded  a 
body  movement  factor.  Many  of  the  variables  that  loaded  on  the  business/office  environment 
factor  in  the  incumbent  data  were  not  rated  by  analysts;  thus  is  it  not  surprising  that  this  factor  is 
not  present.  The  variables  in  the  body  positioning  factor  in  the  analyst  data  load  in  the  physical 
activity  and  manual  work  factor  in  the  incumbent  data, 

Additional  Analyses 

A review  of  the  data  analyses  led  to  additional  questions  and  hypotheses  which  were  not 
planned  beforehand.  Our  taxonomic  model  was  developed  from  research  spanning  many  diverse 
fields.  Information  about  work  context  is  widely  collected,  but  there  is  a lack  of  a coherent  or 
cohesive  theory.  Thus,  additional  exploratory  analyses  are  warranted.  Reported  below  are  a few 
analyses  conducted  to  further  investigate  similarities  and  differences  between  occupations  based 
on  the  work  context  variables. 

Cluster  Analysis 

After  showing  that  work  context  variables  are  capable  of  differentiating  between 
occupations,  it  was  of  interest  to  examine  how  well  the  variables  group  occupations.  A 
hierarchical  cluster  analysis  was  conducted  on  the  37  occupations  in  this  sample  based  on  the 
mean  incumbent  ratings  of  the  work  context  variables.  This  cluster  analysis  identifies 
homogeneous  groups  of  cases  or  variables  and  is  appropriately  used  for  medium-sized  files  (less 
than  200  cases  or  variables).  For  the  current  analysis,  the  unweighted  pair-group  method  using 
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arithmetic  averages  was  used  to  combine  clusters.  It  defines  the  distance  between  two  clusters  as 
the  average  of  the  distances  between  all  pairs  of  cases  in  which  one  member  of  the  pair  is  from 
each  of  the  clusters.  The  squared  Euclidean  distance,  the  sum  of  the  squared  differences  over  all 
of  the  variables,  was  the  distance  measure  in  this  analysis. 

As  presented  in  cluster  analysis  dendogram  (reported  in  Figure  7-7),  the  results  of  the 
analysis  yield  readily  identifiable  and  meaningful  groupings  of  occupations.  Five  job  clusters 
consisting  of  at  least  three  occupations  were  identified,  as  were  five  clusters  consisting  of  only 
two  occupations  each.  Two  occupations  did  not  cluster  with  any  other  occupation. 

The  first  group  consists  of  eight,  primarily  clerk  and  secretarial,  occupations.  Two 
occupations,  Tellers  and  Computer  Programmers,  are  not  an  obvious  match  for  this  group  of 
clerical  jobs,  but  the  requirements  and  tasks  are  somewhat  similar  in  terms  of  equipment, 
attention  to  detail,  need  for  accuracy,  and  physical  job  demands.  The  second  job  group  consists 
of  Education  Administrators,  First  Line  Supervisors,  General  Managers  and  Top  Executives,  and 
non-retail  Salespersons.  These  jobs  require  a significant  amount  of  communication  and  social 
contact.  The  next  group  of  jobs  all  involve  teaching  (Preschool  and  Elementary  School  Teachers, 
and  Teachers’  Aides  and  Assistants).  This  is  an  obvious  grouping  of  jobs,  and  each  job  occurs  in 
the  same  type  of  environment  and  has  very  similar  work  activities.  The  fourth  job  group  consists 
of  Registered  Nurses,  Medical  Assistants,  and  Nursing  Aides.  Again,  this  is  an  obvious  grouping 
of  jobs  that  require  similar  activities  and  are  conducted  in  the  same  type  of  environment.  The 
fifth  group  consists  of  seven  jobs  that  are  somewhat  diverse.  Three  jobs,  Waiters  and  Waitresses, 
Cooks,  and  Food  Preparation  Workers  occur  within  similar  environments,  but  Retail 
Salespersons,  Stock  Clerks,  Cashiers,  and  Janitors  perform  work  in  a different  context.  However, 
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each  of  these  jobs  involve  Providing  a Service  to  Others  ( Descriptor  #6c)  and  involve  similar 
physical  demands,  such  as  being  on  one’s  feet  much  of  the  time.  Standing  (Descriptor  #22b). 

The  cluster  analysis  also  yielded  five  job  clusters  consisting  of  two  jobs  each.  The  two 
jobs  within  the  cluster  are  obviously  related,  but  the  clusters  are  not  closely  linked  to  other  job 
groupings.  Mechanical  and  Chemical  Engineers  form  a small  job  cluster.  Similarly,  Medical  and 
Clinical  Laboratory  Technologists  and  Technicians  form  a distinct  job  group,  as  do  Police  and 
Detective  Supervisors  and  Police  Patrol  Officers.  Maintenance  Repairers  and  Packaging  and 
Filling  Machine  Operators  cluster  together,  and  Truck  Drivers  and  Bus  Drivers  also  form  a 
distinct  cluster.  Finally,  two  jobs.  Machinists  and  Earth  Drillers,  do  not  cluster  with  any  other 
jobs  based  on  the  work  context  variables. 

The  results  of  the  cluster  analysis  are  interesting  given  the  nature  of  the  work  context 
variables.  Based  solely  on  the  physical  environment,  interpersonal  relations,  and  the  structural 
characteristics  of  the  occupation,  meaningful  groupings  of  jobs  can  be  created.  The  jobs  within 
the  clusters  are  similar  in  terms  of  the  work  context  variables,  but  further  examination  of  the  jobs 
within  the  clusters  provides  additional  evidence  of  the  importance  of  the  work  context  variables. 
Without  examining  actual  job  duties.  Generalized  Work  Activities,  or  requisite  skills  and 
education,  the  work  context  variables  yielded  rational  groupings  of  occupations  which  would 
seem  to  perform  similar  activities  and  require  similar  skills. 

Exempt  vs.  Non-Exempt  Jobs 

It  was  expected  that  exempt  jobs  would  differ  from  non-exempt  jobs  in  terms  of  many  of 
the  work  context  variables.  Although  exempt  status  can  differ  between  organizations  even  for  the 
same  job  title,  we  rationally  grouped  the  occupations  as  exempt  or  non-exempt.  Nine  occupations 
were  classified  as  exempt,  and  28  were  classified  as  non-exempt.  Based  on  this  grouping,  t-tests 
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were  conducted  on  variables  hypothesized  to  vary  based  on  the  duties  and  requirements  generally 
found  with  exempt  jobs  (reported  in  Table  7-17a).  In  order  to  provide  an  adequate  sample  size 
and  to  eliminate  the  effect  of  unequal  sample  sizes  within  the  occupations,  the  responses  of  four 
individuals  randomly  selected  from  each  occupation  were  included  in  this  analysis. 

As  expected,  exempt  jobs  involved  significantly  more  Supervising  and  Developing 
Others  (Descriptor  #6a),  Coordinating  or  Leading  Activities  (Descriptor  #6g),  and  Persuading 
and  Influencing  Others  (Descriptor  #6b)  than  did  non-exempt  jobs.  Surprisingly,  being 
Responsible  for  the  Work  of  Others  (Descriptor  #8)  was  not  rated  significantly  higher  by  exempt 
workers,  but  the  difference  is  in  the  expected  direction.  It  was  hypothesized  that  non-exempt 
positions  would  involve  significantly  more  work  in  various  Environmental  Conditions 
(Descriptor  #15).  Non-exempt  jobs  require  the  worker  to  work  Outdoors  (Descriptors  #12c, 
#12d),  in  Vehicles  (Descriptors  #12e,  #12f),  exposed  to  Extreme  Temperatures  (Descriptor 
#15b),  Poor  Lighting  (Descriptor  #15c)  and  Contaminants  (Descriptor  #15d),  and  in  a Cramped 
Work  Space  (Descriptor  #15e)  significantly  more  frequently  than  do  exempt  jobs.  Additionally, 
non-exempt  incumbents  are  exposed  to  job  hazards  (Radiation  [Descriptor  #16a].  Diseases 
[Descriptor  #17a],  High  Places  (Descriptor  #18a).  Hazardous  Conditions  (Descriptor  #19a), 
Hazardous  Equipment  (Descriptor  #20a),  and  Hazardous  Situations  (Descriptor  #2 la)) 
significantly  more  frequently  than  workers  in  exempt  jobs. 

It  was  expected  that  Body  Positioning  (Descriptor  #22)  would  differ  between  exempt  and 
non-exempt  jobs.  Workers  in  exempt  jobs  spent  significantly  more  time  Sitting  (Descriptor  #22a) 
than  did  those  in  non-exempt  jobs,  and  non-exempt  jobs  require  significantly  more  Bending  and 
Twisting  of  the  Body  (Descriptor  #22h),  use  of  hands  to  manipulate  Tools  (Descriptor  #22g),  and 
Making  Repetitive  Motions  (Descriptor  #22i).  Although  the  differences  are  not  significant,  non- 
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exempt  jobs  involve  more  frequent  Kneeling  or  Stooping  (Descriptor  #22e)  and  Keeping  or 
Regaining  Balance  (Descriptor  #22f)  than  do  exempt  jobs. 

It  was  also  hypothesized  that  the  non-exempt  jobs  would  involve  a higher  Level  of 
Automation  (Descriptor  #29),  more  Unstructured  work  (Descriptor  #35),  and  be  more  dependent 
on  a Machine  Driven  Work  Pace  (Descriptor  #39).  These  hypotheses  were  not  supported.  This 
may  be  due  to  the  type  of  occupations  included  in  this  sample.  There  are  only  a few  non-exempt 
jobs  in  the  sample  that  require  a significant  amount  of  work  with  machinery  or  in  a production 
setting.  Most  of  the  non-exempt  jobs  are  performed  in  an  office  or  retail  environment  and  would 
not  be  dependent  on  any  type  of  manufacturing  or  production  machinery  that  gives  considerable 
structure  to  work. 

Working  as  Part  of  a Team 

Given  the  use  of  work  teams  across  a wide  range  of  occupations,  we  explored  differences 
in  jobs  based  on  the  importance  of  working  as  part  of  a team.  High  and  low  teamwork  groups 
were  created  based  on  the  mean  ratings  for  the  occupations.  Those  occupations  rated  at  least  a 
half  a standard  deviation  above  the  mean  in  terms  of  the  importance  of  working  as  a team 
member  were  compared  with  occupations  rated  at  least  a half  a standard  deviation  below  the 
mean.  Eleven  occupations  fall  into  the  high  teamwork  group,  and  1 3 occupations  are  in  the  low 
teamwork  group.  Specifically,  t-tests  were  conducted  on  the  interpersonal  relationship  and 
structural  occupational  characteristic  variables  we  hypothesized  to  be  related  to  teamwork 
(reported  in  Table  7-17b).  As  before,  responses  of  four  individuals  from  each  occupation  were 
randomly  selected  for  use  in  this  analysis. 

Not  surprisingly,  occupations  where  working  as  part  of  a team  is  important  are  found  to 
require  significantly  more  Face-to-Face  Communications  With  Individuals  (Descriptor  #2a)  and 
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Groups  and  to  require  more  Public  Speaking  (Descriptor  #2c)  than  do  jobs  in  which  teams  are 
not  an  important  component.  Also,  high  teamwork  jobs  involve  communication  of  information 
which  is  more  Subjective  (Descriptor  #3)  than  do  low  teamwork  jobs.  It  was  hypothesized  that 
the  occupations  where  teamwork  is  important  would  require  less  communication  by  methods 
other  than  Face-to-Face  talking  (Descriptors  #2a,  #2b).  If  team  members  work  in  the  same  area, 
the  need  to  use  a Telephone  (Descriptor  #2f),  Voice  Mail  (Descriptor  #2e),  Letters  and  Memos 
(Descriptor  #2j),  Handwritten  Notes  (Descriptor  #2i),  or  Electronic  Mail  (Descriptor  #2h)  should 
be  less.  Indeed,  jobs  low  in  teamwork  used  these  communication  methods  more  frequently  than 
did  jobs  high  in  teamwork,  but  the  differences  were  not  significant. 

As  expected,  the  various  types  of  role  relationships  are  more  important  to  jobs  in  which 
working  as  part  of  a team  is  highly  important.  Supervising  and  Developing  Others  (Descriptor 
#6a),  Persuading  Others  (Descriptor  #6b),  Providing  a Service  (Descriptor  #6c),  Taking  a 
Position  Opposed  (Descriptor  #6d)  to  coworkers,  and  Leading  or  Coordinating  Activities 
(Descriptor  #6g)  were  all  rated  as  significantly  more  important  to  high  teamwork  jobs  than  to  low 
teamwork  jobs.  Also,  high  teamwork  employees  are  somewhat  more  Responsible  for  the  Work  of 
Others  (Descriptor  #8)  and  must  handle  Conflict  Situations  (Descriptor  #9)  more  frequently  than 
do  workers  in  low  teamwork  jobs,  but  the  differences  are  not  significant. 

Decisions  made  by  groups  are  likely  to  affect  a larger  number  of  people  (at  least  as  many 
people  as  there  are  in  the  group)  than  do  decisions  made  by  individuals.  As  expected,  Impact  of 
Decisions  (Descriptor  #25)  and  Consequence  of  Error  (Descriptor  #24)  are  rated  significantly 
higher  for  high  teamwork  jobs.  Frequency  of  Decisions  (Descriptor  #25b)  was  also  higher,  but 
not  significantly,  for  high  teamwork  jobs.  Additionally,  it  was  expected  that  occupations  high  in 
teamwork  would  require  more  Working  With  Distractions  (Descriptor  #38)  and  involve  less 
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Unstructured  Tasks  (Descriptor  #35)  and  goals  than  would  jobs  low  in  teamwork.  Again,  these 
differences  were  not  significant,  but  were  in  the  expected  direction. 

Conclusions 

Before  discussing  the  conclusions  derived  from  this  part  of  the  study,  it  is  important  to 
note  the  limitations  associated  with  the  current  research.  The  analyses  presented  above  are  based 
on  a sample  of  37  occupations,  with  as  few  as  four  raters  within  a particular  occupation. 
Although  a wide  range  of  occupations  were  sampled,  this  is  certainly  a small  sample  when 
compared  to  the  entire  world  of  work,  and  caution  is  warranted  when  attempting  to  generalize 
these  findings  to  other  occupations.  Further,  given  the  small  number  of  raters  within  some 
occupations,  it  is  not  possible  to  say  definitively  that  these  data  necessarily  reflect  work  context 
factors  found  in  these  occupations  across  the  economy. 

It  is  possible  that  substantive  changes  could  occur  in  the  relationships  and  structure  of  the 
results  with  a larger  sample.  Also,  the  focus  of  these  analyses  was  to  provide  evidence  for  the 
internal  structure  of  the  variables.  It  will  be  important  for  future  research  to  focus  on  the 
relationships  of  these  variables  with  other  measures  outside  the  work  context  domain. 
Nonetheless,  the  current  findings  provide  evidence  for  the  measurement  of  these  variables  and 
have  valuable  implications  for  further  research  and  practice. 

Overall,  the  work  context  variables  can  be  reliably  assessed  by  job  incumbents  and 
analysts.  However,  the  nature  of  some  work  context  variables  may  make  it  difficult,  if  not 
impossible,  to  accurately  make  ratings  based  solely  on  task  lists.  Actual  job  performance  or  at 
least  visual  observation  of  the  job  being  performed  would  be  required  to  accurately  respond  to 
many  descriptors.  The  moderate  levels  of  agreement  for  the  mean  ratings  between  incumbents 
and  occupational  analysts  may  be  due  to  how  salient  and  pervasive  the  work  context  variables  are 
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to  job  incumbents.  It  is  often  difficult  to  get  an  accurate  assessment  of  these  variables  simply  by 
reading  task  lists.  In  terms  of  describing  occupations,  incumbent  data  are  more  likely  to  provide 
appropriate  work  context  information. 

One  concern  is  the  somewhat  marginal  results  for  interrater  agreement  with  as  many  as 
eight  descriptors.  Further  review  of  these  items  is  needed  in  order  to  evaluate  their  utility  in  the 
taxonomy.  Using  the  average  number  of  raters  across  occupations,  there  were  five  items  with 
interrater  agreement  coefficients  below  .50.  When  the  agreement  is  estimated  for  30  raters, 
obviously  these  coefficients  increase,  but  a few  items  continue  to  have  insufficient  interrater 
agreement.  Revision,  or  removal,  of  the  items  may  be  warranted  if  raters  are  unable  to  provide 
reliable  results. 

The  patterns  of  results  across  occupations  provide  evidence  that  the  variables  can 
differentiate  between  occupations.  The  comparison  of  variables  across  the  six  example 
occupations  provided  evidence  showing  occupations  differed  in  rational,  measurable  ways. 
Further,  the  discriminant  analysis  provided  evidence  that  groupings  of  work  context  variables  can 
be  used  to  effectively  differentiate  between  occupations. 

We  are  also  very  encouraged  about  the  capability  of  the  work  context  descriptors  alone  to 
provide  meaningful  information  about  the  similarities  and  differences  among  occupations,  as 
well  as  how  those  occupations  are  positioned  in  the  workforce.  Thus,  the  very  positive  results  we 
obtained  from  our  studies  of  job  clusters,  exempt  versus  non-exempt  status,  and  teamwork 
clearly  emphasize  the  role  that  work  context  Variables  will  have  in  future  research  and 
applications  with  the  0*NET  content  model.  Given  the  effects  of  work  context  variables  on 
workers  and  work  performance,  analyzing  only  work  tasks,  behaviors,  and  outcomes  is  not 
adequate,  work  context  also  must  be  examined  and  integrated  with  other  job  analysis  information 
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in  order  to  provide  complete  information  about  occupations.  Measuring  work  context  more 
systematically  will  contribute  to  a better  understanding  of  work  and  will  provide  valuable 
information  essential  for  a comprehensive  Occupational  Information  System. 

An  interesting  outcome  of  the  analyses  is  the  prospect  of  reevaluating  the  structure  of  the 
taxonomy  based  on  the  results  of  the  factor  and  discriminant  analyses.  The  item  loadings  on  the 
factors  and  functions  were  highly  consistent  with  the  grouping  in  the  original  taxonomy,  but 
there  were  several  items  which  loaded  with  descriptors  from  other  higher-order  factors.  For 
instance,  the  factor  associated  with  managerial  activities  is  made  up  of  variables  from  both  the 
interpersonal  and  structural  components  of  the  taxonomy.  Findings  such  as  this  are  not  all  that 
surprising  given  the  nature  of  the  items,  but  future  research  might  investigate  the  need  to 
reconfigure  the  taxonomy. 

There  has  been  tremendous  fragmentation  and  lack  of  standardization  in  previous 
attempts  to  assess  aspects  of  the  context  in  which  work  occurs.  This  is  likely  due  to  the  extremely 
broad  range  of  constructs  that  may  be  considered  under  the  rubric  of  work  context.  Our 
taxonomy  builds  on  previous  work  to  present  a set  of  meaningful  work  context  variables  which 
accurately  differentiates  between  occupations  and  provides  a framework  for  further  study  of  the 
structure,  reliability,  and  validity  of  work  context  measures. 
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Figure  7-1 

Work  Context  and  Work  Processes 
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Figure  7-2 

Work  Context  Items  - Job  Analysis  Instruments  Matrix 
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Figure  7-2  (continued) 

Work  Context  Items  - Job  Analysis  Instruments  Matrix 
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Figure  7-3 

Work  Context  Taxonomy:  Higher  Order  Factors 


g?4 


Figure  7-4 

Work  Context  Taxonomic  Structure 
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Figure  7-4  (continued) 

Work  Context  Taxonomic  Structure 
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Figure  7-4  (continued) 

Work  Context  Taxonomic  Structure 
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Mgure  /- 4 (continued) 

Work  Context  Taxonomic  Structure 
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Figure  7-5 

Impact  of  Work  Context  Variables 
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Figure  7-6 

Occupations  Included  in  Data  Analysis 


Occupation 

Code 

Occupation  Title 

Incumbent  Sample 
Size 

15005 

Education  Administrators 

9 

19005 

General  Managers  & Top  Executives 

55 

22114 

Chemical  Engineers 

5 

22135 

Mechanical  Engineers 

5 

25105 

Computer  Programmers 

9 

31303 

Teachers,  Preschool 

4 

31305 

Teachers,  Elementary  School 

13 

32502 

Registered  Nurses 

30 

32902 

Medical  & Clinical  Laboratory  Technologists 

5 

32905 

Medical  & Clinical  Laboratory  Technicians 

6 

49008 

Salespersons,  Except  Scientific  & Retail 

7 

49011 

Salespersons,  Retail 

21 

49021 

Stock  Clerks,  Sales  Floor 

15 

49023 

Cashiers 

28 

51002 

First  Line  Supervisors,  Clerical  & Administrative 

57 

53102 
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53121 
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10 
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55108 
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78 
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General  Office  Clerks 
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11  1 
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Police  Patrol  Officers 

25 
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Waiters  & Waitresses 

22 
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Cooks,  Restaurant 

6 
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Food  Preparation  Workers 

25 

66005 

Medical  Assistants 
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66008 

Nursing  Aides,  Orderlies  & Attendants 

17 
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Janitors  & Cleaners 

30 
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Maintenance  Repairers,  General  Utility 

37 
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Earth  Drillers,  Except  Oil  & Gas 

8 

89108 

Machinists 

4 
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Packaging  & Filling  Machine  Operations 

12 

97102 

Truck  Drivers,  Heavy  or  Tractor  Trailer 

16 

97111 

Bus  Drivers,  Schools 

16 
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Figure  7-7 

Cluster  Analysis  Dendoeram  of  Job  Groupings 
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Table  7-1  (continued) 

Dgscriptive  Statistics  Across  All  Occupations  and  Reliability  Estimates  for  Rated  Differences  Between  Occupations:  Work  Context 
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19c.  Extent  of  Injury  0 = No  treatment  required  0.86  0.89  0.31  0.88 

7 = Injury  resulting  in  permanent  total 
impairment/death 


Table  7-1  (continued) 

Descriptive  Statistics  Across  All  Occupations  and  Reliability  Estimates  for  Rated  Differences  Between  Occupations:  Work  Context 
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Table  7-1  (continued) 

Descriptive  Statistics  Across  All  Occupations  and  Reliability  Estimates  for  Rated  Differences  Between  Occupations:  Work  Context 
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Table  7-2 

Reliability  of  Rated  Differences  Between  Occupations  Considering  Varying  Numbers  of  Raters: 
Work  Context 


Variable 


Descriptor 

ria 

o 

cr 

1. 

Communication  Formality 

.18 

.87 

2. 

Communication  Methods 

2a. 

Face-to-Face  Individuals 

.08 

.72 

2b. 

Face-to-Face  Groups 

.26 

.92 

2c. 

Public  Speaking 

.28 

.92 

2d. 

Video  Conference 

.03 

.44 

2e. 

Voice  Mail 

.47 

.96 

2f . 

Telephone 

.55 

.97 

2g- 

Interactive  Computer 

.22 

.90 

2h. 

Electronic  Mail 

.47 

.96 

2i. 

Handwritten  Notes 

.29 

.93 

2j- 

Letters  and  Memos 

.24 

.91 

2k. 

Written  Reports 

.26 

.91 

3. 

Communication  Subjectivity 

.07 

.69 

4. 

Social  Interaction 

.26 

.91 

5. 

Privacy  of  Communication 

.22 

.90 

6. 

Job  Interactions 

6a. 

Supervise/Develop  Others 

.33 

.94 

6b. 

Persuade  or  Influence 

.43 

.96 

6c. 

Provide  Service  to  Others 

.28 

.92 

6d. 

Take  Opposing  Position 

.26 

.91 

6e. 

Work/Contribute  to  Team 

.13 

.82 

6f. 

Deal  With  Public 

.34 

.94 

6g- 

Coordinate/Lead  Activity 

.20 

.88 

7. 

Health/Safety  of  Others 

.61 

.98 

8. 

Responsible  Others'  Work 

.34 

.94 

9. 

Conflict  Situations 

.24 

.90 

10. 

Unpleasant  Individuals 

.16 

.86 

11. 

Physical  Aggression 

.53 

.97 

12. 

Work 

Settings 

12a. 

Indoors,  Controlled 

.36 

.94 

12b. 

Indoors,  Uncontrolled 

.29 

.93 

12c. 

Outdoors,  Exposed 

.71 

.99 

12d. 

Outdoors,  Covered 

.53 

.97 

12e. 

Open  Vehicle/Equip. 

.56 

.97 

12f. 

Enclosed  Vehicle/Equip. 

.75 

.99 

13. 

Privacy  of  Work  Area 

.39 

.95 

14. 

Physical  Proximity 

.22 

.90 

15. 

Environmental  Conditions 

O 


702 


Table  7-2  (continued) 

Reliability  of  Rated  Differences  Between  Occupations  Considering  Varying  Numbers  of  Raters: 
Work  Context 


Descriptor 

15a.  Distracting  Noise 
15b.  Extreme  Temperature 
15c.  Poor  Lighting 
15d.  Contaminants 
15e.  Cramped  Work  Space 
15f.  Whole  Body  Vibration 

16.  Radiation 

16a.  Exposure 
1 6b.  Likelihood  of  Inj  ury 
16c.  Extent  of  Injury 

17.  Diseases/Infections 
17a.  Exposure 

17b.  Likelihood  of  Injury 
17c.  Extent  of  Injury 

18.  High  Places 
18a.  Exposure 

18b.  Likelihood  of  Injury 
1 8c.  Extent  of  Injury 

19.  Hazardous  Conditions 
19a.  Exposure 

19b.  Likelihood  of  Injury 
19c.  Extent  of  Injury 

20.  Hazardous  Equipment 
20a.  Exposure 

20b.  Likelihood  of  Injury 
20c.  Extent  of  Injury 

21.  Hazardous  Situations 
21a.  Exposure 

21b.  Likelihood  of  Inj  ury 
21c.  Extent  of  Injury 

22.  Body  Positioning 
22a.  Sitting 
22b.  Standing 

22c.  Climbing  Ladders,  etc. 

22d.  Walking  or  Running 
22e.  Kneeling  o Crouching 
22f.  Keeping/!,  ^gaining  Balance 
22g.  Handling  Tools,  Objects 
22h.  Bending/Twisting  Body 


Variable 


ri 

^30 

.18 

.87 

.54 

.97 

.37 

.95 

.42 

.96 

.29 

.93 

.61 

.98 

.23 

.90 

.13 

.82 

.09 

.76 

.52 

.97 

.49 

.97 

.48 

.96 

.59 

.98 

.53 

.97 

.47 

.96 

.44 

.96 

.43 

.96 

.43 

.96 

.66 

.98 

.66 

.98 

.59 

.98 

.51 

.97 

.51 

.97 

.38 

.95 

.75 

.99 

.74 

.99 

.58 

.98 

.58 

.98 

.41 

.95 

.26 

.91 

.25 

.91 

.49 

.97 

Table  7-2  (continued) 

Reliability  of  Rated  Differences  Between  OccupationsConsidering  Varying  Numbers  of  Raters: 
Work  Context 


Numbers  of  Raters  on  Each  Variable 


Descriptor 

n 

T30 

22i.  Making  Repetitive  Motions 

.28 

.92 

23. 

Work  Attire 

23a.  Business/Office  Clothes 

.67 

.98 

23b.  Special  Uniform 

.56 

.97 

23c.  Maintenance  Clothes 

.63 

.98 

23d.  Common  Safety  Attire 

.69 

.99 

23 e.  Special  Safety  Attire 

.45 

.96 

24. 

Consequence  of  Error 

.45 

.96 

25. 

Impact  of  Decisions 
25a.  Level  of  Decisions 

.34 

.94 

25b.  Frequency  of  Decisions 

.33 

.94 

26. 

Accountable  for  Results 

.27 

.92 

27. 

Decision  Latitude 

.28 

.92 

28. 

Frustrating  Circumstances 

.19 

.87 

29. 

Level  of  Automation 

.21 

.89 

30. 

Task/Performance  Clarity 

.05 

.62 

31. 

Accuracy/Exactness 

.22 

.89 

32. 

Details  and  Completeness 

.14 

.83 

33. 

Constant  Awareness 

.23 

.90 

34. 

Repetitive  Activities 

.22 

.89 

35. 

Unstructured  Tasks/Goals 

.23 

.90 

36. 

Level  of  Competition 

.13 

.81 

37. 

Deadlines/Time  Pressure 

.16 

.85 

38. 

Work  With  Distractions 

.14 

.83 

39. 

Machine  Driven  Pace 

.27 

.92 

Note.  Statistics  are  based  on  37  occupations  with  Work  Context  Questionnaire  responses  from  at 
least  4 incumbents  (mean  number  of  incumbents  = 19.68;  median  = 12.5;  harmonic 
mean  = 9.44). 

aSingle  rater  estimates  of  reliability  were  obtained  by  calculating  the  intraclass  correlation  for 
single-judge  ratings  across  occupations:  ICC(1,1)  = fBMS-WMSl/BMS  + (k-l)WMSl  (Shrout  & 
Fleiss,  1979). 

bEstimates  of  reliability  for  30  raters  were  obtained  by  applying  the  Spearman-Brown  correction 
formula  to  the  single  rater  reliability  estimates. 


Table  7-4 

Analysis  of  Variance  for  Descriptor.  Occupation,  and  Relevant  Interactions  as  Sources  of 
Variation:  Work  Context 


Source  of  Variation 

SS 

df 

MS 

F 

Occupations 

9367.69 

36 

260.21 

12.18* 

S(Occupations) 

14763.67 

691 

21.37 

Descriptor 

60466.24 

89 

679.40 

301.04* 

Descriptor  x Occupations 

53764.99 

3204 

16.78 

7.44* 

Descriptor  x S (Occupations) 

138793.68 

61499 

2.26 

Note,  Occupations  are  treated  as  random,  between-subjects  effects,  while  descriptors  are  treated 
as  fixed,  within-subjects  effects. 

*E  < .05 


Table  7-5 

Interrater  Agreement  Coefficients:  Work  Context 


Number  of  Raters  on  Each  Variable jj  r30 

Descriptor J57 ^40  .95 

Note.  Statistics  are  based  on  37  occupations  with  Work  Context  Questionnaire  responses  from  at 
least  4 incumbents  (mean  number  of  incumbents  = 19.68,  median  = 12.5,  harmonic  mean  = 

9.44).  Full  sample  interrater  agreement  coefficients  (&)  were  obtained  by  considering  “Descriptor 
x Occupations”  terms  from  Table  7-4  as  true  variance.  Error  variance  was  defined  as  the 
“Descriptor  x S(Occupations)  term.  Estimates  of  reliability  for  30  raters  were  obtained  by 
applying  the  Spearman-Brown  correction  formula  to  the  k rater  reliability  estimates,  where  k is 
the  harmonic  mean  of  the  number  of  raters  for  each  occupation. 
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Table7-9a 

Intercorrelations  of  Descriptors  (Occupation  Level  Data):  Work  Context— Interpersonal  Relationships 

— Descriptor 1.  2a.  2b.  2c.  2d.  2e.  2f.  2g.  2h. 

1 . Communication  Formality  — 

2.  Communication  Methods 
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Table  7-9a  (continued) 

Intercorrelations  of  Descriptors  (Occupation  Level  Data):  Work  Context-  Interpersonal  Relationships 
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Table  7-9b 

Intercorrelations  of  Descriptors  (Occupation  Level  Data):  Work  Context-Physical  Work  Conditions 
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Table  7-9b  (continued) 

Intercorrelations  of  Descriptors  (Occupation  Level  Data):  Work  Context— Physical  Work  Conditions 
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Table  7-9b  (continued) 

Intercorrelations  of  Descriptors  (Occupation  Level  Data):  Work  Context-Physical  Work  Conditions 
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Table  7-9b  (continued) 

Intercorrelations  of  Descriptors  (Occupation  Level  Data):  Work  Context-Physical  Work  Condition 
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Table  7-9c 

Intercorrelations  of  Descriptors  (Occupation  Level  Data):  Work  Context-  Structural  Job  Characteristics 
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Table  7-  10a  (continued) 

Intercorrelations  of  Descriptors  (Individual  Level  Data):  Work  Context--Interpersonal  Relationships 
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Table  7- 10b 

.Intercorrelations  of  Descriptors  (Individual  Level  Data):  Work  Context-Physical  Work  Conditions 
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Table  7- 10b  (continued) 

Intercorrelations  of  Descriptors  (Individual  Level  Data):  Work  Context-  Physical  Work  Conditions 
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Table  7- 10b  (continued) 

Intercorrelations  of  Descriptors  (Individual  Level  Data):  Work  Context-Physical  Work  Conditions 
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Table  7- 10b  (continued) 

Intercorrelations  of  Descriptors  (Individual  Level  Data):  Work  Context-Physical  Work  Conditions 
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Table  7- 10b  (continued) 

Intercorrelations  of  Descriptors  (Individual  Level  Data):  Work  Context-Physical  Work  Conditions 


O 

cn 

(N 


TD 

cn 

CN 

d 

cn 

CN 


X> 

CO 

CN 

cd 

co 

CN 


CN 

CN 


X 

CN 

CN 


W> 

CN 

CN 


CN 

CN 


O 

CN 

CN 


CN 

CN 


O 

CN 

CN 


X> 

CN 

CN 


cd 

CN 

CN 


O 
■*— » 
cx 

• 

U* 

o 

0) 

Q 


00 

C 


ON 


00  oo 
Tt  co 


oo  co  on 
CN  ^ CO 


o\  t O 
— ■ vo  ^ cn 


VO  Os  O VO  CN 
r-  in  cn  cn 


cn 

CN 

I 


o 


5/3  c 
o 

00  W 


<D 

T3 

T3 


00 

c 

oo  *2 

c X2 

G o 
2 
o 


oo 

c 


u 


o 

H 


c .S 


CO 

O 

CX 

So 

T3 

O 


§>=5  s -a 


c 

E 5 


00  g> 


o 

o 


VO 
cn 

>* 

T3 

O 

D3 

a».g 


•■3  c « 

+-*  cd  cd 

3j  w U S£  W 

cd  x5  d *d  d 

__  CN  CN  CN  CN  CN 

CQ  CN  CN  CN  CN  CN 

CN 

CN 


cd 

oo 
&> 

f)  g» 

•all  € 
« § s « 

<D  *— i Cd  *5p 

* CQ  . O 

«4-I  oo 
CN  CN 
CN  CN 


£ 

c 

*3 

c 

o 

ffl 


x .X 
CN  CN 
CN  CN 


cx 

<D 

DS 

00 
c 

1 
s 


00 

cn 


VO 
Tf  Tf 


1 

CN 

00 

CN 

O 

1 

CN 

O 

1 

Ti- 

ON 

VO 

CN 

en 

1 

cn 

t 

1 

1 

1 

On 

On 

r- 

CN 

O 

1 

O 

O 

! as 

r- 

< 

00 

VO 

cn 

CN 

1 

cn 

CN 

CN 

CN 

1 

1 

CN  ON 

VO 

c- 

CN 

{ 

VO 

Tf  Tf 

CN 

1 

CN 

cn 

CN 

VO  ON 

CN 

r- 

O 

00 

00 

CN 

V0  CN 

CN 

1 

CN 

CN 

Tf 

cn 

r- 

r- 

VO 

00 

r» 

cn 

vo  cn 

CN 

1 

CN 

CN 

cn 

cn  cn 

vo 

r- 

Tf 

Tf 

CN 

Tf  CN 

CN 

1 

r-H 

CN 

CN 

vo  cn 

cn 

O 

r- 

ON 

cn  ^ 

1 

O 

tT 

cn 

Tf 

ON 

cn  cn 

CN 

Tf 

CN 

n* 

CN 

^ CN 

cn 

1 

cn 

CN 

0 

h O OO 

cn  *— < cn 
00  1 1 
Q> 

X 


O 

X 


w 

C P 

o 

'X  2» 

a < 


o 

£ 

cn 

CN 


On  ^ 
CN  — < 

i i 

E 

’5  E 
< 5 

<§  5? 

« ,0 

C CO 

o _ 

£ .2 

SO 

o 

■H  O- 

U 00 

cd  X)  d *d  d 
cn  cn  cn  cn  cn 
CN  CN  CN  CN  CN 


O <L) 

O f 

» § u 
_ 2 « 
O c § 
^ ID  g 

gas 

.3  O C 
CO  y *2 

3 cx  42 
CQ  co  S 


cd 

3 

T3 


*3 

.5 

C 

o 

TD 

<D 

c/5 

cd 

X) 

T3 

O 

jd 

"3 

"E 

o 

C/5 

C 

to 

•4—* 

cd 

7> 

fa  T3 

O 

o £2 

< o 

^ 2 

g S3 

O co 

1 1 

Ij 

o Q 

r-  X 

cn  o 

o, 

2 ‘C 
x o 

o 00 
cd  a> 
CD  T3 

E ■§ 
2 § 


c 

o 

• H 
W 

cd 

a, 

3 


E 

o 

T3 

C 

2 o 
o 

cd  O 

5 g 

o •- 
00 

•3  « 

*°  c 
«2  o 

£ "o 
•2  c 
£ .2? 

3 Vi 

g S3 

Tf  3) 
w c 
00  *3 

Tf  cd 


II 

2 

d 

o 

2 


c 

<u 

X) 

E 

2 

o 

c 


O 

X* 


CJ5 

<r? 


flj 

o 

2 

cb 

-3 

U 

3 

_o 

w 

CB 

Q 

3 

O 

O 


u 

2 

So 


X 

V 

3 

o 

u 

X 

o 


cb 

3 

TD 


•o 

3 


C/3 

Lri 

o 

*-» 

o 

*C 

o 

co 

4> 

Q 


C/3 

c 

o 

2l 

^ o 

4>  O 

— L* 

JS  4> 

CB  *3 

H £ 


On 

co 


00 

co 


r- 

co 


vo 

CO 


wo 

CO 


CO 


CO 

CO 


<N 

CO 


CO 


o 

CO 


ON 

<N 


00 

cs 


r- 

cs 


VO 

(N 


-O 

wo 

(N 


03 

wo 

CS 


cs 


o 

a* 


o 

Q 


VO 

<S 


co  r- 
CO  ^ 


VO  N Tt 
<S  O <N 


O^tMO 

(N  O ^ ^ 


o on  co  vo 
^ O N ^ 


ON  CO  CO  co  Os  O 
CO  O CS  CS  O <N 


VO  ^ CO  CO  *-«  vo  wo 
M CO  O O r)  M 


^^vOTfoorfinh 

OOCSMOO^ON 


^^tvoocovovo^oo 
fOMO^^O^OO 


^WOCS^VOONVOOWOON 

OMM^H(NOO^OTf 


VOONTfCNcOxtrfOsTfwowo 

COOO^(S^ri^Mco(S 


VO^CJvOvOOn^’IOVO'hOCS 


-"-wocOOONOr-CSO^frvoOCS 
^tCOCS^NCOCS^^CNCNCNM 


OVOC^TtTtCO^NTt^fOOOCOO 

(nr^nOO^MMOO^^CS^ 


ONCOTtcOCNOsOsOsOOOOOVOCSOr^ 

<n>nMco^^^^o)o^^^NO 


ON 

CO 


PQ 


3 .S3 
o o 

— a.) 

S3  Q 
o 

D OJ  ^ 

g Q ® 

4)  W— i 4) 
3 O > 
a"  a> 
u u J 
52  «s  . 
£ g-  cfl 

r9  £ W0 

U h cs 

wo  - 
es <N 


c/5  Q 


irj 
CS  CO 

C/3 

C 

o 

u *s 
« 3 

C/3 

4> 

o * 

>o  o 
o c2 

c 

5 o 
32 
Z S 
£ § 

J9  8 
wo  p 

<N  < 


r-r-wo^-^ON(Swococovor-r- 

OM^OCONCO^OfHfH^^H 


C/3 

4> 

o 

c 

CB 

*-*  c 

E *2 

3 CB 

2 E 

G 1 
S?< 


2 « o 

% is  t? 

T*\  QJ 


u 

5 l 

tt.  J 


O CO 
tn 

u a> 

g 5 

i S 

It 

« p 

Q,  CB 

fer*  u< 

22  ^ 
CO  p 
CB  p 

H < 


CO 

CO 

D 

c 

4) 

”5  60 

*3-  8 

E I 

o b 

u g 
13 

c ^ 

CB  *g 

.52  « 

2 c 
4>  O 

Q U 


CB 

O 

O 

co 

4)  co 

\s 

•— < co 
> CB 

I- 


3 

o 


; 3 cl  cb 

«->  i-. 


p a> 

& E 


3 

O 4) 
w O 
O CB 
CB  Ph 

*-  C 
S3  o 

Q .2: 


S g o .5 


CO  ^ 

4)  ?- 
C 


5 Q 
£ 


4)  3 

&.  52 

4)  3 

0$  D 


4) 

c 

j*  •£ 

b o 

O CB 


vpr-oooNO^cScoTtwovor^‘odoN 

CSCSCSCSCOCOCOCOCOCOCOCOCOCO 


W) 

c 


3 

4) 

-9 


CB 

3 

*2 

*> 

•o 


c 

o 

•o 

4) 

CO 

CB 

x> 

T3 

OJ 

JS 

3 

o 

3 

o 

co 

c 

o 


co 

c 

o 

a. 

3 


o 2 
r-  -a 
co  c 

. , o 

° 2 

O ^ 
CB  co 

® 73 
E E 
p o 

^ 4) 

eQ 


c t- 


o 

a, 


cb  £ 
u ~C 

CB  8 
T3  -2 

S 73 

o -g 

JU  g 

2 3 


C CB 

a, 
«g  3 

E ° 

i 8 

3 3 

Tt  > 

^ W) 
00  ^ 

3 
O 


II 


4) 

3 

.£P 

co 

CB 


o 

ERIC 


fKP 

i> 


Table  7-1  la 

Principal  Components  Analysis  Pattern  Matrix:  Work  Context 


Factors 


Descriptor 

FI 

F2 

F3 

F4 

F5 

F6 

F7 

Communalitv 

1.  Communication  Formality 

-.02 

-.23 

.28 

-.10 

.51 

.45 

-.15 

.63 

2.  Communication  Methods 

2a.  Face-to-Face  Individuals 

-.03 

.33 

.33 

.22 

.35 

.18 

.19 

.46 

2b.  Face-to-Face  Groups 

.15 

.01 

.73 

-.14 

.27 

.16 

.09 

.68 

2c.  Public  Speaking 

-.05 

.13 

.30 

-.01 

.15 

.01 

.71 

.63 

2d.  Video  Conference 

-.09 

.10 

.26 

-.15 

-.03 

.13 

.53 

.41 

2e.  Voice  Mail 

.02 

-.64 

.46 

-.14 

.37 

-.19 

.04 

.81 

2f.  Telephone 

-.11 

-.41 

.20 

.07 

.59 

-.02 

.24 

.63 

2g.  Interactive  Computer 

-.27 

-.28 

.10 

.35 

.29 

.06 

.10 

.38 

2h.  Electronic  Mail 

-.18 

-.56 

.35 

-.06 

.27 

-.20 

-.09 

.59 

2i.  Handwritten  Notes 

-.07 

-.47 

-.11 

-.16 

.56 

.12 

.28 

.67 

2j.  Letters  and  Memos 

-.23 

-.42 

-.02 

-.22 

.58 

-.08 

.13 

.64 

2k.  Written  Reports 

.35 

-.20 

.24 

.14 

.18 

.67 

.16 

.74 

3.  Communication  Subjectivity 

-.04 

.29 

.20 

-.57 

.17 

.07 

.20 

.53 

4.  Social  Interaction 

-.05 

.16 

.06 

.00 

.52 

.22 

.55 

.65 

5.  Privacy  of  Communication 

.21 

.22 

.66 

-.03 

-.19 

.20 

.13 

.62 

6.  Job  Interactions 

6a.  Supervise/Develop  Others  .09 

.12 

.68 

-.09 

.04 

.20 

.07 

.55 

6b.  Persuade  or  Influence 

.03 

-.11 

.76 

-.08 

.06 

-.17 

.46 

.84 

6c.  Provide  Service  to  Others 

.05 

.01 

.41 

.51 

.25 

-.01 

.43 

.67 

6d.  Take  Opposing  Position 

.05 

-.12 

.78 

-.27 

-.11 

-.07 

.04 

.72 

6e.  Work/Contribute  To  Team-.05 

.42 

.46 

.18 

.20 

.20 

.00 

.50 

6f.  Deal  With  Public 

.07 

-.04 

.20 

.09 

.26 

.08 

.81 

.79 

6g.  Coordinate/Lead  Activity 

.13 

.12 

.71 

-.22 

.09 

.18 

.22 

.67 

7.  Health/Safety  of  Others 

.43 

.45 

.10 

-.33 

-.36 

.50 

.02 

.88 

8.  Responsible  Others'  Work 

.21 

.22 

.66 

-.03 

-.19 

.20 

.13 

.62 

9.  Conflict  Situations 

.51 

-.16 

.33 

-.21 

.23 

.30 

.30 

.67 

10.  Unpleasant  Individuals 

.34 

-.07 

-.08 

-.15 

.31 

.45 

.53 

.73 

11.  Physical  Aggression 

.43 

.13 

.04 

-.27 

.05 

.70 

.13 

.79 

12.  Work  Settings 

12a.  Indoors,  Controlled 

-.79 

-.09 

.06 

.00 

.02 

-.21 

-.09 

.69 

12b.  Indoors,  Uncontrolled 

.36 

-.02 

.30 

-.22 

-.48 

-.17 

-.10 

.54 

• 12c.  Outdoors,  Exposed 

.86 

.06 

.01 

-.21 

-.04 

.27 

.20 

.91 

12d.  Outdoors,  Covered 

.72 

-.07 

.13 

-.11 

-.22 

.09 

.26 

.67 

12e.  Open  Vehicle/Equip. 

.87 

.22 

.17 

.08 

.10 

.00 

-.16 

.88 

12f.  Enclosed  Vehicle/Equip.  .77 

-.27 

.09 

-.11 

-.08 

.26 

.37 

.90 

13.  Privacy  of  Work  Area 

-.23 

-.53 

.49 

-.28 

.25 

.04 

-.02 

.72 

14.  Physical  Proximity 

.12 

.52 

.06 

.20 

.14 

.53 

.03 

.63 

15.  Environmental  Conditions 

15a.  Distracting  Noise 

.72 

.00 

-.08 

-.02 

-.21 

-.12 

-.09 

.59 
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Table  7-1  la  (continued) 

Principal  Components  Analysis  Pattern  Matrix:  Work  Context 


Factors 


Descriptor 

FI 

F2 

F3 

F4 

F5 

F6 

F7 

Communalitv 

15b.  Extreme  Temperature 

.91 

.10 

-.01 

-.05 

-.26 

.01 

.10 

.91 

15c.  Poor  Lighting 

.91 

-.02 

-.02 

.10 

-.08 

.18 

.18 

.91 

15d.  Contaminants 

.59 

.16 

.10 

.17 

-.52 

.12 

-.23 

.74 

15e.  Cramped  Work  Space 

.79 

.26 

.16 

.16 

-.25 

.15 

-.05 

.83 

15f.  Whole  Body  Vibration 

.84 

.15 

.01 

.16 

.05 

.03 

-.06 

.75 

16a.  Radiation— Exposure 

.22 

-.03 

.35 

.25 

.07 

-.01 

-.45 

.44. 

17a  Diseases/Infections— 

-.21 

.30 

-.01 

.17 

-.12 

.81 

.02 

.84 

Exposure 

18a.  High  Places—  Exposure 

.80 

.27 

.25 

-.04 

.05 

-.22 

-.10 

.84 

19a.  Hazardous  Conditions— 

.62 

.14 

.20 

.38 

-.35 

.20 

-.35 

.87 

Exposure 

20a.  Hazardous  Equipment— 

.77 

.09 

.15 

.03 

-.42 

-.05 

-.19 

.84 

Exposure 

21a.  Hazardous  Situations— 

.55 

.41 

.03 

-.05 

-.53 

.08 

-.08 

.77 

Exposure 

22.  Body  Positioning 

22a.  Sitting 

-.09 

-.93 

-.08 

.09 

.15 

.07 

-.11 

.93 

22b.  Standing 

.04 

.93 

.04 

-.03 

-.21 

-.01 

.05 

.92 

22c.  Climbing  Ladders,  etc. 

.71 

.35 

.25 

-.09 

-.06 

-.32 

-.27 

.87 

22d.  Walking  or  Running 

.14 

.83 

.01 

-.24 

-.11 

.06 

.02 

.78 

22e.  Kneeling  or  Crouching 

.19 

.83 

.00 

-.20 

.01 

.07 

.02 

.76 

22f.  Keeping/Regain  Balance .47 

.79 

.13 

-.06 

.07 

.05 

-.02 

.88 

22g.  Handling  Tools,  Objects  .40 

.50 

-.27 

.47 

-.38 

-.17 

.06 

.88 

22h.  Bending/Twisting  Body 

.34 

.82 

-.09 

-.08 

-.12 

-.03 

.21 

.87 

22i.  Making  Repet.  Motions 

.07 

.45 

-.37 

.60 

-.16 

-.17 

-.04 

.76 

23.  Work  Attire 

23a.  Business/Office  Clothes-.32 

-.47 

.13 

.02 

.61 

-.16 

.17 

.77 

23b  Special  Uniform 

.06 

.19 

.10 

.25 

-.33 

.62 

.37 

.75 

23c.  Maintenance  Clothes 

.55 

.30 

-.12 

.00 

-.58 

-.27 

-.12 

.84 

23d.  Common  Safety  Attire 

.48 

.19 

.32 

.14 

-.55 

.26 

-.28 

.83 

23e.  Special  Safety  Attire 

.72 

.21 

.28 

.00 

-.06 

.17 

-.18 

.72 

24.  Consequence  of  Error 

.36 

-.24 

.42 

.31 

-.15 

.55 

-.17 

.81 

25.  Level  of  Decisions 

25a.  Impact  of  Decisions 

.18 

-.32 

.56 

.25 

.15 

.54 

.08 

.82 

25b.  Frequency  of  Decisions 

.26 

-.30 

.40 

.35 

.24 

.38 

.32 

.75 

26.  Accountable  for  Results 

.20 

-.31 

.62 

.36 

-.22 

.27 

.24 

.83 

27.  Decision  Latitude 

.31 

-.46 

.67 

.16 

-.12 

.05 

.03 

.79 

28.  Frustrating  Circumstances 

.40 

-.48 

.55 

.12 

-.19 

-.01 

-.11 

.75 

29.  Level  of  Automation 

-.22 

-.25 

.20 

.72 

.02 

.03 

-.27 

.74 

30.  Task/Performance  Clarity 

-.12 

.20 

-.47 

.27 

.09 

.27 

-.05 

.44 
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Table  7-1  la  (continued) 

Principal  Components  Analysis  Pattern  Matrix:  Work  Context 


Factors 


Descriptor 

FI 

F2 

F3 

F4 

F5 

F6 

F7 

Communalitv 

3 1 . Accuracy/Exactness 

-.18 

-.25 

-.02 

.68 

-.03 

.33 

-.05 

.67 

32.  Details  and  Completeness 

-.14 

-.13 

.01 

.76 

.02 

.32 

.08 

.73 

33.  Constant  Awareness 

.43 

.11 

.17 

.41 

.00 

.47 

.27 

.69 

34.  Repetitive  Activities 

.10 

.13 

-.40 

.80 

.13 

.10 

-.02 

.85 

35.  Unstructured  Tasks/Goals 

-.05 

-.12 

.37 

-.70 

.30 

.08 

.02 

.75 

36.  Level  of  Competition 

.25 

.17 

.52 

.16 

-.25 

-.18 

.24 

.54 

37.  Deadlines/Time  Pressure 

.25 

-.17 

-.02 

.37 

-.59 

.20 

.11 

.63 

38.  Work  With  Distractions 

.10 

-.54 

.14 

.41 

.09 

.09 

.14 

.53 

39.  Machine  Driven  Pace 

.32 

-.09 

-.14 

.69 

-.36 

-.11 

-.11 

.76 

Percent  of  Variance 

23 

16 

10 

8 

6 

4 

3 

Eigenvalue 

18.37 

12.55  8.05  6.65  4.84 

3.25 

2.68 

Note.  N = 37.  The  Correlation  Matrix  was  based  on  means  calculated  at  the  occupation  level. 
These  loadings  are  based  on  an  orthoganal  varimax  rotation.  FI  = Environmental  Conditions, 
F2  = Physical  Activity  and  Manual  Work,  F3  = Managerial/Interpersonal  Relations,  F4  = 
Structured/Machine  Operations,  F5  = Business/Office  Environment,  F6  = Health  and  Safety 
Conditions,  F7  = Interacting  with  Public. 
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Table  7- lib 

Confirmatory  Factor  Analysis  Pattern  Matrix:  Work  Context 


Factors 


Descriptor 

FI 

F2 

F3 

Communalitv 

1 . Communication  Formality 

-.12 

.49 

.29 

.34 

2.  Communication  Methods 

2a.  Face-to-Face  Individuals 

.09 

.31 

-.01 

.10 

2b.  Face-to-Face  Groups 

.18 

.73 

.10 

.58 

2c.  Public  Speaking 

.00 

.49 

-.06 

.24 

2d.  Video  Conference 

.01 

.42 

-.10 

.19 

2e.  Voice  Mail 

-.26 

.57 

.48 

.62 

2f.  Telephone 

-.35 

.45 

.41 

.50 

2g‘.  Interactive  Computer 

-.29 

.07 

.46 

.30 

2h.  Electronic  Mail 

-.37 

.34 

.43 

.44 

2i.  Handwritten  Notes 

-.41 

.39 

.29 

.40 

2j.  Letters  and  Memos 

-.57 

.37 

.19 

.50 

2k.  Written  Reports 

.39 

.41 

.43 

.50 

3.  Communication  Subjectivity 

-.07 

.51 

-.50 

.52 

4 .Social  Interaction 

-.12 

.45 

-.05 

.22 

5.  Privacy  of  Communication 

-.22 

.40 

.17 

.24 

6.  Job  Interactions 

6a.  Supervise/Develop  Others 

.26 

.57 

.03  ' 

.40 

6b.  Persuade  or  Influence 

.08 

.71 

.15 

.53 

6c.  Provide  Service  to  Others 

.13 

.28 

.38 

.24 

6d.  Take  Opposing  Position 

.14 

.62 

.07 

.41 

6e.  Work/Contribute  To  Team 

.17 

.28 

-.07 

.12 

6f.  Deal  With  Public 

.03 

.50 

.14 

.27 

6g.  Coordinate/Lead  Activity 

.25 

.72 

-.04 

.59 

7.  Health/Safety  of  Others 

.66 

.17 

-.43 

.65 

8.  Responsible  Others'  Work 

.47 

.46 

-.02 

.43 

9.  Conflict  Situations 

.38 

.64 

.14 

.58 

10.  Unpleasant  Individuals 

.21 

.45 

.05 

.25 

11.  Physical  Aggression 

.48 

.38 

-.10 

.39 

12.  Work  Settings 

12a.  Indoors,  Controlled 

-.72 

-.09 

.02 

.52 

12b.  Indoors,  Uncontrolled 

.43 

.09 

-.11 

.21 

12c.  Outdoors,  Exposed 

.76 

.29 

-.10 

.67 

12d.  Outdoors,  Covered 

.67 

.24 

.03 

.51 

12e.  Open  Vehicle/Equip. 

.79 

.12 

-.05 

.65 

12f.  Enclosed  Vehicle/Equip. 

.64 

.37 

.21 

.60 

13.  Privacy  of  Work  Area 

-.36 

.60 

.35 

.61 

14.  Physical  Proximity 

.35 

.07 

-.15 

.15 

15.  Environmental  Conditions 

15a.  Distracting  Noise 

.62 

-.11 

-.04 

.39 
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Table  7-1  lb  (continued) 

Confirmatory  Factor  Analysis  Pattern  Matrix:  Work  Context 


Factors 


Descriptor 

FI 

F2 

F3 

Communalitv 

15b.  Extreme  Temperature 

.85 

.03 

-.10 

.74 

15c.  Poor  Lighting 

.83 

.10 

.11 

.70 

15d.  Contaminants 

.79 

-.23 

.01 

.68 

15e.  Cramped  Work  Space 

.91 

.00 

-.03 

.82 

15f.  Whole  Body  Vibration 

.75 

.00 

.02 

.56 

16a.  Radiation-Exposure 

.27 

.00 

.26 

.14 

17a.  Diseases/Infection— 

.16 

-.02 

.01 

.03 

Exposure 

18a.  High  Places— Exposure 

.71 

.17 

-.19 

.57 

19a.  Hazardous  Conditions— 

.83 

-.22 

.19 

.78 

Exposure 

20a.  Hazardous  Equipment— 

Exposure 

.83 

-.09 

-.04 

.70 

21a.  Hazardous  Situations— 

.76 

-.17 

-.33 

.72 

Exposure 

22.  Body  Positioning 

22a.  Sitting 

-.36 

.04 

.76 

.71 

22b.  Standing 

.35 

-.14 

-.73 

.67 

22c.  Climbing  Ladders,  etc. 

.66 

.06 

-.31 

.54 

22d.  Walking  or  Running 

.34 

.01 

-.76 

.69 

22e.  Kneeling  or  Crouching 

.35 

.04 

-.72 

.65 

22f.  Keeping/Regaining  Balance  .60 

.10 

-.58 

.71 

22g.  Handling  Tools,  Objects 

.59 

-.58 

-.20 

.73 

22h.  Bending/Twisting  Body 

.50 

-.07 

-.70 

.74 

22i.  Making  Repetitive  Motions 

.24 

-.69 

-.09 

.54 

23.  Work  Attire 

23a.  Business/Office  Clothes 

-.60 

.36 

.39 

.65 

23b.  Special  Uniform 

.42 

.05 

.12 

.19 

23c.  Maintenance  Clothes 

.65 

-.38 

-.33 

.68 

23d.  Common  Safety  Attire 

.78 

-.08 

.04 

.62 

23e.  Special  Safety  Attire 

.77 

.18 

-.05 

.63 

24.  Consequence  of  Error 

.56 

.19 

.56 

.66 

25.  Impact  of  Decisions 

25a.  Level  of  Decisions 

.30 

.50 

.61 

.71 

25b.  Frequency  of  Decisions 

.29 

.45 

.60 

.64 

26.  Accountable  for  Results 

.41 

.36 

.60 

.66 

27.  Decision  Latitude 

.35 

.43 

.58 

.65 

28.  Frustrating  Circumstances 

.40 

.29 

.53 

.53 

29.  Level  of  Automation 

-.06 

-.27 

.66 

.51 

30.  T ask/Performance  Clarity 

-.06 

-.41 

-.03 

.17 

Table  7-1  lb  (continued) 

Confirmatory  Factor  Analysis  Pattern  Matrix:  Work  Context 


Factors 


Descriptor 

FI 

F2 

F3 

Communalitv 

3 1 . Accuracy/Exactness 

.00 

-.28 

.65 

.50 

32.  Details  and  Completeness 

.07 

-.25 

.60 

.43 

33.  Constant  Awareness 

,60 

.13 

.28 

.46 

34.  Repetitive  Activities 

.17 

-.59 

.32 

.48 

35.  Unstructured  Tasks/Goals 

-.22 

.73 

-.22 

.63 

36.  Level  of  Competition 

.43 

.23 

.01 

.24 

37.  Deadlines/Time  Pressure 

.48 

-.29 

.35 

.44 

38.  Work  With  Distractions 

.03 

.08 

.70 

.50 

39.  Machine  Driven  Pace 

.45 

-.57 

.42 

.70 

Percent  of  Variance 

23 

16 

10 

Eigenvalue 

18.37 

12.55 

8.05 

Note.  N = 37.  The  Correlation  Matrix  was  based  on  means  calculated  at  the  occupation 
level.  These  loadings  are  based  on  an  orthoganal  varimax  rotation.,  FI  = Physical  Work 
Conditions,  F2  = Interpersonal  Relationships,  F3  = Structural  Occupation  Characteristics. 
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Table  7-12 

Descriptor  Means  and  Standard  Deviations  on  Six  Example  Occupations:  Work  Context 


C3 
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Table  7-12  (continued) 

Descriptor  Means  and  Standard  Deviations  on  Six  Example  Occupations:  Work  Context 


ea 

l-i 

<D 

C 

<D 

o 


<D 

o 

#* 

as 

r-* 

00 

CO 

vo 

o 

vn 

On 

2.00 

vo 

Ov 

00 

00 

CO 

O 

CO 

X 

G 

03 

G 

§ 

*-*— * 
D 

r- 

cn 

VO 

vo 

ON 

n- 

CN 

ON 

o 

ON 

o 

00 

o 

ON 

TC 

cm’ 

ON 

p 

CN 

°n 

p 

p 

CM* 

o 

CM* 

CN 

CM* 

p 

p 

<D 

*s 

13 

II 

.s 

CD 

<D 

OS 

Ut 

D 

3 

SI 

m 

VO 

ON 

oo 

CO 

00 

VO 

On 

CO 

CM 

CN 

vo 

On 

00 

*s 

G 

CO 

oo 

p 

°°. 

ON 

r- 

O 

p 

p 

CN 

p 

o 

On 

p 

s 

CD 

o 

CM 

vo 

CO 

o 

vo 

CO 

CM 

CM 

CO 

CM 

vo 

CO 

CO 

SD 

cs 

CO 

CO 

r- 

o 

< 

CN 

X 

ON 

CM 

r-* 

Tl* 

On 

CO 

vn 

O 

X 

00 

00 

cS 

W 

oo 

CO 

p 

CO 

p 

p 

p 

4—4 

00 

p 

vn 

X 

On 

o 

C/5 

l-i 

l-( 

D 

o 

m 

X 

CN 

CM 

CM 

CM 

CM 

O 

CM 

CM 

CM* 

X 

X 

CM* 

o 

C 

4— < 

cd 

II 

*S 

D 

3 

SI 

CO 

o 

o 

r- 

o 

O 

O 

CO 

o 

o 

CO 

X 

a 

0 

OV 

VO 

o 

vo 

O 

p 

o 

vo 

CO 

p 

p 

p 

On 

CM 

vo 

CO 

CM 

vo 

CM 

CM 

o 

o 

CO 

CM 

CO 

o 

vo 

o 

Ql 

( 

o 

X 

( 

CO 

CN 

VO 

Tf 

r- 

r- 

CM 

00 

CN 

Tt 

i-i 

& 

Tj“ 

M; 

CN 

vn 

VO 

O) 

00 

p 

o 

On 

4—4 

O) 

CN 

X 

CN 

cd 

Oh 

1-H 

D 

*n 

CN 

coi 

o 

CN 

o 

o 

o 

CM 

o 

CM 

CO 

O 

CM 

CM* 

CM* 

CN 

CM* 

<D 

u 

X 

II 

.2 

H-l  _N, 

o 3 

SI 

o 

00 

X 

00 

00 

vo 

o 

Tf 

CM 

CM 

00 

O 

00 

00 

O 

00 

00 

X 

00 

o 

00 

CM 

ON 

p 

oo 

^-4 

p 

OO 

vo 

vo 

p 

o 

p 

p 

o 

X 

On 

00 

Oh 

CO 

vo 

CO 

CM 

CO 

CM 

CO 

vd 

CO 

CM 

vd 

CM 

CO 

vn 

CO 

CM* 

*3 

SD 

00 

o 

ON 

( 

( 

CN 

| 

CO 

CM 

CO 

O 

o 

On 

00 

X 

CN 

X 

CO 

a 

V 

CA 

V 

C/5 

l-i 

O 

m 

VO 

vq 

CM 

CM 

vo 

o 

o 

ON 

d 

CO 

p 

CM 

4—4 

CM 

p 

On 

O 

CM 

CM 

p 

CM 

p 

CO 

CN 

CN 

CN 

CO 

CN* 

4-J 

C/5 

II 

• —4 

3 

OX) 

<D 

OS 

*7  Cl 

SI 

o 

r- 

o 

o 

r- 

o 

CO 

r- 

r- 

CO 

CO 

r- 

O 

r- 

CO 

o 

o 

CO 

o 

CN 

p 

cn 

00 

vo 

o 

4—4 

r- 

p 

p 

00 

o 

On 

o 

o 

CO 

CO 

vo 

CM 

CM 

vd 

o 

o 

CO 

vo 

O 

CM* 

Vh 

c« 

t-N 

SD 

00 

oo 

0.44 

r- 

r- 

00 

o 

CO 

CO 

o 

000 

o 

o 

r- 

CN 

X 

O 

o 

CO 

0.44 

<D 

4— * 

3 

D 

E 

at 

vo 

VO 

p 

CM 

ON 

d 

00 

o 

o 

o 

p 

CM 

CO 

o 

o 

o 

o 

o 

o 

o 

P 

p 

1—4 

X 

O 

O 

o 

H-H 

CO 

o 

CD 

E 

II 

E 

o 

a 

Ut 

OX) 

3 

SI 

CO 

CO 

CM 

00 

vo 

o 

CM 

o 

O 

o 

o 

On 

o 

CO 

O 

CO 

l 

CN 

u 

o 

CO 

CO 

CM 

vo 

o 

CM 

o 

O 

o 

o 

p 

p 

p 

O 

CO 

CN 

1-4 

Oh 

CM 

CM 

o 

CO 

o 

vd 

o 

o 

o 

o 

o 

CO 

CO 

o 

o 

o 

o 

< 

SD 

00 

2.04 

r— 4 

VO 

vo 

vo 

r- 

CO 

o 

o 

vn 

CO 

vn 

CN 

CN 

r- 

o 

o 

tj- 

w 

i-i 

flj 

CD 

<D 

> 

(n 

KT) 

vq 

ON 

o 

p 

VO 

o 

r- 

o 

00 

o 

P 

CM 

CM 

ON 

CO 

o 

CN 

CM 

o 

CM 

p 

CN 

p 

o 

CO 

CM 

On 

O 

OX) 

O 

3 

II 

03 

H 

O 

C 

D 

3 

SI 

ON 

— < 

oo 

CO 

^—4 

00 

vo 

CM 

00 

o 

1—1 

ON 

CM 

OO 

00 

00 

X 

X 

X 

i-H 

a 

S 

X 

W 

CO 

00 

CO 

IO 

vo 

CO 

CM 

CM 

ON 

o 

vd 

p 

C- 

o 

o 

o 

p 

CO 

vo 

ON 

CO 

X 

CM 

On 

O 

o 

X 

X 

o 

o 

4— > 

CD 

*c 

u 

C/5 

<D 

Q 


3 

On 

X 


> 

*3 

u 

< 

T3  . 
c3 


^ ^ <D  C 
^ ^ .2 


cd 

<D 

Q 


VO 


o 

o 

U 


0 

1 
S 


c3 
OX) 

vo  3C 


<-i 

o 
£ 

e 52 

<D  G 
X O 

O Cd 
£ 2 
£ K 

00  4-» 

C O 

B,£ 

oo  C 

<d  o 
OS  U 


C/5 

13 

3 C 
T3  O 

• — H 

C/5 
CO 

P 


T3 

<D 

td  a 
o o 


> 

-3 

£ 


oa 

oo 

< 


§ 

a s 

o .a 

& & 

c J3 

D On 


a 

o 

c/5  U 

bo  •* 

.s  ^ 

X o 

Z o 

CO 

* 
o 

£ 


G 

o 

o 

c 

D 


T3  T3 

S £ 


T3  T3 
<D  CD 
to  w 
O 44 

cu  > 
X r9 
W U 

CO  w 

1-4 

O O 
O O 
•O  *3 

3 3 

O o 


a*  §* 

*5  69 

cr  <d 

{2  13 


^ IS  < 

T?  CD  . , — 1 


X 

(D 

> 

T3 

<D 

C/5 

O 


03  X 
CN  CN 


d *3 

CM  CM 


o 

X 

CD 

> 

C 

o (§ 


CD  (4-4 
CM  CN 


C?  O 

6U 
X "* 

o 


^ >5  ■*-* 

^ g C 


o 

>> 

o 

03 

> 

*u 

Oh 


u, 

eu 

3 

O 


C/5  - 

> 

X G 
Oh  W 


2 

3 

<d  2 

.2  s 

*0  CD 


X J-h 
X * 

Q ffl 

03  X 


0X) 

c/5 

.£ 

a 

*4—4 

C3 

X 

c 

OX) 

J 

E 

C3 

1-4 

o 

4—* 

G 

o 

O 

Cu 

U 

ci 

d 

X 

<D 

O 

03 

CD 

CO 

X 

Lh 

O 


E 

cS 


<D 


cm 

lO 

i> 


H 

LO 

TNo 


C^  00  On  O CM 


o Tt  m 


Table  7-12  (continued) 

Descriptor  Means  and  Standard  Deviations  on  Six  Example  Occupations:  Work  Context 
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Table  7-12  (continued) 

Descriptor  Means  and  Standard  Deviations  on  Six  Example  Occupations:  Work  Context 
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Table  7-12  (continued) 

Descriptor  Means  and  Standard  Deviations  on  Six  Example  Occupations:  Work  Context 
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Table  7-13 

Rotated  Correlations  Between  Discriminating  Variables  and  Canonical  Discriminant  Functions: 
Work  Context 


Descriptor 

FI 

F2 

F3 

F4 

F5 

F6 

IF2 

_2 

Tl 

1.  Communication  Formality 

-.04 

.14 

.01 

.04 

.06 

.11 

.05 

.14 

2a.  Face-to-Face  Individuals 

-.02 

.01 

.08 

-.06 

.05 

.06 

.02 

.09 

2b.  Face-to-Face  Groups 

.05 

.08 

• .12 

.09 

.22 

.08 

.09 

.19 

2c.  Public  Speaking 

.06 

-.02 

.15 

-.20 

.19 

-.02 

.10 

.20 

2d.  Video  Conference 

.02 

.02 

.05 

-.02 

.05 

.01 

.01 

.06 

2e.  Voice  Mail 

-.20 

.20 

-.07 

.06 

.24 

.20 

.19 

.33 

2g.  Interactive  Computer 

-.13 

.07 

.02 

-.05 

-.06 

.11 

.05 

.16 

2f.  Telephone 

-.26 

.22 

-.02 

.00 

.06 

.29 

.20 

.39 

2k.  Written  Reports 

.05 

.21 

.08 

-.02 

-.04 

.17 

.08 

.18 

2h.  Electronic  Mail 

-.21 

.14 

-.05 

.07 

.17 

.17 

.13 

.33 

2i.  Handwritten  Notes 

-.15 

.15 

-.02 

-.01 

-.02 

.12 

.07 

.21 

2j.  Letters  and  Memos 

-.16 

.07 

-.07 

.01 

.00 

.08 

.06 

.17 

3.  Communication  Subjectivity  .05 

-.04 

.04 

.06 

.09 

.01 

.03 

.08 

4.  Social  Interaction 

-.02 

.05 

.16 

-.14 

.07 

.12 

.07 

.18 

5.  Privacy  of  Communication 

-.11 

.04 

.04 

.21 

.14 

.05 

.08 

.16 

6a.  Supervise/Develop  Others 

.06 

.03 

.18 

.16 

.27 

.03 

.16 

.23 

6b.  Persuade  or  Influence 

.02 

.03 

.17 

-.02 

.51 

-.01 

.31 

.30 

6c.  Provide  Service  to  Others 

.02 

.03 

.16 

-.12 

.17 

.05 

.14 

.19 

6d.  Take  Opposing  Position 

.04 

.06 

.08 

.14 

.28 

.01 

.12 

.18 

6e.  Work/Contribute  To  Team 

.04 

-.01 

.11 

.05 

.10 

.02 

.04 

.11 

6f.  Deal  With  Public 

.02 

.09 

.17 

-.24 

.17 

.08 

.14 

.24 

6g.  Coordinate/Lead  Activity 

.05 

.03 

.10 

.06 

.26 

.04 

.09 

.15 

7.  Health/Safety  of  Others 

.36 

.03 

.20 

.20 

.03 

-.13 

.26 

.45 

8.  Responsible  Others'  Work 

.09 

.03 

.12 

.19 

.27 

-.05 

.14 

.23 

9.  Conflict  Situations 

.08 

.16 

.10 

.01 

.18 

.04 

.08 

.17 

10.  Unpleasant  Individuals 

.08 

.10 

.13 

-.10 

.03 

.05 

.06 

.13 

11.  Physical  Aggression 

.21 

.22 

.26 

.07 

-.06 

.15 

.23 

.38 

12a.  Indoors,  Controlled 

-.22 

-.11 

.09 

.08 

.05 

.07 

.09 

.25 

12b.  Indoors,  Uncontrolled 

.15 

-.03 

-.08 

.15 

.13 

-.03 

.07 

.20 

12c  Outdoors,  Exposed 

.46 

.24 

-.09 

-.13 

.12 

.01 

.39 

.55 

12d.  Outdoors,  Covered 

.29 

.11 

-.09 

-.05 

.12 

-.06 

.12 

.38 

12e.  Open  Vehicle/Equip. 

.30 

.09 

-.21 

.01 

.04 

.33 

.25 

.40 

12f.  Enclosed  Vehicle/Equip 

.39 

.48 

-.08 

-.15 

.21 

-.09 

.50 

.60 

13.  Privacy  of  Work  Area 

-.15 

.17 

.05 

.19 

.25 

-.04 

.17 

.27 

14.  Physical  Proximity 

.10 

-.01 

.13 

.02 

-.07 

.09 

.04 

.16 

15a.  Distracting  Noise 

.11 

.05 

-.11 

-.01 

-.03 

-.01 

.03 

.14 

15b.  Extreme  Temperature 

.34 

.10 

-.17 

-.07 

.00 

-.02 

.16 

.39 

15c.  Poor  Lighting 

.22 

.15 

-.09 

-.14 

-.01 

.08 

.10 

.26 

15d.  Contaminants 

.23 

.03 

-.06 

.18 

-.16 

.01 

.12 

.29 

15e.  Cramped  Work  Space 

.21 

.02 

-.06 

.01 

-.02 

.12 

.06 

.20 

O 
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Table  7-13  (continued) 

Rotated  Correlations  Between  Discriminating  Variables  and  Canonical  Discriminant  Functions: 
Work  Context 


15f.  Whole  Body  Vibration 

.31 

.15 

-.27 

-.05 

rj 

.02 

ro 

-.07 

.20 

VT 

.45 

1 6a.  Radiation 

.07 

.02 

-.04 

.14 

.03 

.16 

.06 

.16 

17  a.  Diseases/Infections 

.15 

.02 

.36 

.15 

-.28 

.08 

.27 

.37 

18a.  High  Places 

.31 

-.05 

-.22 

.08 

.17 

.39 

.34 

.43 

19a.  Hazardous  Conditions 

.25 

.05 

-.09 

.20 

-.04 

.07 

.14 

.30 

20a  Hazardous  Equipment 

.40 

.12 

-.23 

.10 

.05 

.13 

.33 

.50 

21a.  Hazardous  Situations 

.32 

-.02 

.02 

.05 

-.02 

.03 

.13 

.36 

22a.  Sitting 

-.40 

.49 

-.22 

-.01 

-.09 

-.10 

.47 

.61 

22b.  Standing 

.37 

-.48 

.23 

-.01 

.05 

.05 

.44 

.59 

22c.  Climbing  Ladders,  etc. 

.27 

-.12 

-.27 

.14 

.18 

.29 

.30 

.42 

22d.  Walking  or  Running 

.25 

-.28 

.15 

.11 

-.06 

.10 

.21 

.42 

22e.  Kneeling  or  Crouching 

.19 

-.21 

.03 

.05 

-.07 

.15 

.11 

.28 

22f.  Keeping/Regaining  Balance 

.18 

-.11 

.04 

.05 

.00 

.11 

.06 

.18 

22g.  Handling  Tools,  Objects 

.14 

-.10 

-.08 

-.10 

-.16 

-.01 

.09 

.18 

22h.  Bending/Twisting  Body 

.25 

-.23 

.07 

-.03 

-.06 

.05 

.13 

.34 

22i.  Making  Repetitive  Motions 

.03 

-.12 

-.08 

-.18 

-.24 

-.01 

.14 

.19 

23  a Business/Office  Clothes 

-.41 

.17 

-.02 

-.11 

.23 

.24 

.35 

.51 

23b.  Special  Uniform 

.22 

.09 

.36 

-.11 

-.19 

.05 

.28 

.41 

23c.  Maintenance  Clothes 

.33 

-.20 

-.28 

.12 

-.03 

-.10 

.26 

.47 

23  d Common  Safety  Attire 

.42 

.08 

.03 

.40 

-.12 

.03 

.44 

.53 

23e.  Special  Safety  Attire 

.26 

.06 

-.02 

.11 

-.02 

.30 

.17 

.32 

24.  Consequence  of  Error 

.16 

.24 

.10 

.20 

.06 

-.04 

.17 

.31 

25  a.  Level  of  Decisions 

.05 

.20 

.16 

.09 

.22 

.00 

.13 

.23 

25b.  Frequency  of  Decisions 

.06 

.17 

.19 

-.05 

.20 

-.04 

.12 

.23 

26.  Accountable  for  Results 

.08 

.13 

.11 

.08 

.22 

-.13 

.12 

.19 

27.  Decision  Latitude 

.00 

.18 

.05 

.11 

.28 

.05 

.13 

.20 

28.  Frustrating  Circumstances 

.03 

.12 

-.05 

.11 

.18 

.00 

.06 

.14 

29.  Level  of  Automation 

-.12 

.02 

-.06 

.05 

-.07 

.01 

.07 

.16 

30.  T ask/Performance  Clarity 

-.02 

-.01 

.01 

-.07 

-.11 

-.08 

.02 

.07 

31.  Accuracy/Exactness 

-.08 

.11 

.02 

-.08 

-.09 

.00 

.08 

.16 

32.  Details  and  Completeness 

-.04 

.07 

.03 

-.09 

-.08 

-.04 

.06 

.12 

33.  Constant  Awareness 

.12 

.10 

.13 

-.07 

.04 

.03 

.05 

.16 

34.  Repetitive  Activities 

-.02 

.01 

-.05 

-.20 

-.21 

-.01 

.11 

.16 

35.  Unstructured  Tasks/Goals 

-.05 

.06 

.06 

.15 

.19 

.11 

.13 

.17 

36.  Level  of  Competition 

.06 

.00 

.04 

.03 

.20 

-.05 

.06 

.11 

37.  Deadlines/Time  Pressure 

.06 

.08 

-.02 

.02 

-.11 

-.17 

.06 

.13 

38.  Work  With  Distractions 

-.07 

.12 

.00 

-.03 

.02 

-.03 

.02 

.12 

39.  Machine  Driven  Pace 

.01 

.04 

-.19 

-.08 

-.20 

-.17 

.15 

.19 

Rc 

90 

86 

83 

77 

73 

69 

Percent  of  Variance 

21 

14 

11 

7 

6 

5 

Table  7-13  (continued) 

Rotated  Correlations  Between  Discriminating  Variables  and  Canonical  Discriminant  Functions: 


Work  Context 
Descriptor 

FI 

F2 

F3 

F4 

F5 

F6 

IF2 

if 

Eigenvalue 

4.27 

2.86 

2.16 

1.44 

1.11 

.92 

Note.  Statistics  are  based  on  37  occupations  with  Work  Context  questionnaire,  responses  from  at 
least  4 incumbents  (mean  number  of  incumbents  - 19.68,  median  - 12.5,  harmonic  mean  - 9.44) 
FI  = Environmental  Conditions,  F2  = Physical  Activity,  F3  = Health  Care,  F4  = Interacting  with 
Public,  F5  = Managerial/Interpersonal  Relations,  F6  = Hazardous  Work  Conditions. 

TF  = Sum  of  squared  rotated  standardized  discriminant  function  coefficients  across  6 functions. 
t|2=  Variance  in  Work  Context  Level  Scale  ratings  accounted  for  by  occupations. 

The  statistics  “Rc,”  “Percent  of  Variance,”  and  “Eigenvalue,”  were  calculated  based  on  the 
unrotated  discriminant  functions. 


761 


o 


O 

g 

06 

Uh 

£ 

C/2 

<D 

g 

6 

C/2 

w 


X) 

5 

13 

04 

T3 

G 

g 

C/2 

G 

O 

cd 

a 

G 

o 

o 

o 

< 

C/2 

C/2 

O 

U4 

O 

< 

C/2 

o 


cd 
■*— » 

00 

<D 

> 


r-» 

X 

cd 


4J 

X 

Q 

o 

4— » 

4— » 

G 

C/2 

o 

13 

u 

G 

< 

5— i 

o 

n© 

G 

cd 

G 

G 

o 

a) 

X 

G 

E 

P 

G 

G 

a 

o 

£ 

u 

o 

G 

a) 

o 

£ 
4— » 

G 

a> 

o 

£ 

4) 

OQ 

a> 

OQ 

G 

O 

C/2 

O 

C/2 

o 

c3 

G 

(U 

D 

E 

o 

Uh 

tO 

Sts 

U 

S 

n 

in 

On 

CN 

oo 

On 

N" 

CN 

CN 

CO 

VO 

oo 

VO 

VO 

'Gl 

CO 

CO 

CO 

CO 

r- 

O 

CN 

so 

CN 

N- 

CN 

o 

o 

cd 

d 

d 

d 

d 

CN 

CN 

cd 

d 

d 

d 

d 

a 

uH 

o 

X 

in 

t 

CN 

r- 

SO 

VO 

CN 

o 

o 

VO 

On 

oo 

oo 

O 

l 

oo 

CN 

CO 

VO 

N" 

r- 

CO 

in 

r- 

in 

r- 

N" 

VO 

so 

oo 

r- 

OO 

r- 

r- 

r- 

r- 

r- 

On 

o 

O 

o 

o 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

* 

* 

* 

* 

* 

* 

in 

CO 

CO 

o 

r- 

On 

CO 

CN 

CN 

o 

CO 

o 

o 

QLI 

°0 

p 

Os 

CN 

VO 

SO 

CO 

CO 

CO 

c4 

cd 

CN 

CN 

CN 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

cs 

oo 

CN 

in 

r- 

OO 

so 

VO 

o 

o 

OO 

o 

X 

CN 

4— *1 

o 

as 

oo 

CN 

CN 

so 

r- 

ON 

SO 

o 

SO 

CO 

CO 

vd 

CN 

in 

d 

SO 

cd 

in 

SO 

cd 

CN 

6f\ 

o 

so 

CO 

On 

CO 

N" 

o 

CN 

CN 

oo 

o 

CN 

On 

On 

r- 

oo 

oo 

On 

r- 

ON 

ON 

On 

ON 

OO 

oo 

ON 

On 

ON 

ON 

oo 

oo 

ON 

On 

On 

OO 

OO 

r- 

r- 

oo 

oo 

r- 

r- 

d 

o 

d 

d 

d 

d 

d 

d 

o 

o 

o 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

Q| 

CO 

in 

CN 

On 

r- 

o 

CN 

N- 

On 

r- 

oo 

On 

VO 

N- 

CO 

CO 

N- 

tj- 

X 

M 1 

ool 

o 

in 

CN 

CN 

in 

in 

vo 

vo 

VO 

X 

OO 

r- 

X 

X 

N- 

CO 

CO 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

to 

SI 

in 

SO 

On 

On 

so 

N" 

O 

CO 

OO 

f-H 

r- 

N- 

ON 

oo 

X 

N- 

ON 

vo 

OO 

CO 

o 

G 

in 

^d; 

On 

oo 

so 

in 

CO 

so 

CO 

CO 

N- 

CO 

00 

oo 

VO 

4-H 

CN 

CN 

< 

CN 

CN 

d 

CN 

CN 

d 

cd 

d 

d 

d 

d 

d 

d 

A 

so 

CO 

00 

00 

r- 

CO 

o 

of 

CO 

VO 

vo 

Tf 

X 

r- 

CN 

VO 

r- 

o 

CO 

On 

r- 

oo 

00 

r- 

r- 

OO 

r- 

On 

oo 

r- 

SO 

On 

00 

On 

X 

On 

OO 

oo 

OO 

On 

r- 

VO 

o 

o 

d 

d 

o 

o 

o 

o 

d 

d 

d 

d 

O 

d 

O 

d 

d 

d 

d 

d 

d 

o 

d 

Q| 

X 

l 

so 

in 

On 

o 

CN 

VO 

N" 

VO 

oo 

Nf 

VO 

VO 

CN 

VO 

CO 

OO 

00 

CN 

r- 

ool 

m 

ON 

r- 

O 

oo 

r- 

OO 

CN 

N- 

CO 

X 

CO 

O 

CN 

CN 

ON 

r- 

G 

a> 

X 

d 

d 

d 

d 

o 

d 

d 

d 

CN 

d 

d 

d 

E 

G 

SI 

CO 

CN 

CO 

CO 

CO 

VO 

Td_ 

ON 

VO 

VO 

4— « 

so 

N- 

4-H 

N- 

On 

VO 

VO 

CN 

oo 

X 

CO 

O 

5 

CO 

p 

oo 

in 

oo 

On 

as 

OO 

o 

On 

Os 

CO 

O 

CO 

VO 

CO 

N- 

N" 

CO 

in 

cd 

X 

cd 

CN 

CN 

cd 

cd 

CN 

d 

in 

X 

cd 

CN 

X 

CN 

X 

d 

d 

d 

O 

4— > 

cx 

■ p*^ 

u-, 

o 

C/2 

Q 


4— » 

a 

<D 

X* 

G 

00 

G 
O 
’•*— * 
cd 
o 


G 

.2 
’-4— » 

a 

cd 


C/2 

U* 

0> 

X 


a, 

o 

13 

> 

CD 


C/2 

U* 

<D 

<D  5 
S ° 

<D 

G 
CG 

5 


G hS 

I 

o o 
U co 


C/2  tT 

G a3 
O c/2 

o E 

0 <U 

ft  cx 
B 3 

5 M 


a> 

o 

'E 

cu 

00 


G 

o 


C/2 

o 

Ou 


Ut 

O 
CD 
T3  <D 
cd  u 


o 


ff? 

I 5 
^ -s 

CL 


>4 

4-» 

’> 

o 

< 


O G 

PL  Pl 


o 

o' 

13 


T3  X 

Uh  — 

o 
o 
U 


X cd  X O 
O XXX 


*G  4L 


OX) 

X 


0) 

X 


o 

>% 

4—* 

o 

4— ( 

G 

00 

£ 
4— > 

G 

<D 

DC 


*-l 

O 

£ 

52  ^ 
<u  c 

X o 

6 "3 

z)  3 
X CO 

c/2  ■*—* 

c o 
O X 
Cl  ^ 
U2  G 
a>  O 

06  U 


C/2 

Id 

G G 
T3  O 


C/2 

C/2 

2 

OX) 

OX) 

< 


> 

•3 

£ 

4— * 

G 

3 ]3 

<d  .2 

~G  c/2 

& >> 
x 

D Ol 


-a 

a> 


c 

o 

U 

s«  c/2 
OX)  u. 
C o 

■P4  © 

4_j  CC 
4— » 

O c 

00  x 
o (N 


rt  t/2 

TD  c 
O O 

C/2  X 

o .ti 

CL  -© 
* G 

ffl  o 
rfU 
o 13 
p c 


e 

3 

4-4 

G 

Ut 

a> 

CL  « 


*2  5 is 


G | 
° § 


E 

oo£ 
G ^ 

<D 

E 

o 


OX)  c^2 

G G 
G 
G 


X 

OX) 


C/2 

s 


Ui 

O 

O 

OL 


« © 
a>  § 
o .2 
cd  4-2 

a,  « 

on  ^ 

> 

I'g 

U PQ 
cl  a) 
E o 

G X 

U £ 


cd  fr  x 

4—* 

G 
O 

U 


& 

G 

S’ 

4-. 

o 


TD 

O 

O 

X 


O 

<N 


CO  -rf  X 


oo  on  o — ' (N 


> 

G 

w 

tri 


cd  X o ''O 
>n  >n 


<U  4-4 

in  m 


a 

o 

cd 

-3 

G 

C2 

vd 


£ 

3 

C/2 

o 

CL  yj 

x gS 
W J 


<u 


cd  X 
v©  vo 


er|c 


Cl 

CO 

i> 


73 

<D 

cd 


pp 


X) 

2 

13 

C* 

73 

C 

</3 

c 

.2 

■4— * 

03 

Oi 

D 

CJ 

CJ 

O 

< 

(/3 

<Z3 

o 

U* 

CJ 

< 

</3 

o 


c3 

4—4 

CO 

0> 

> 


<D 

Q 

■4— * 
C/3 

> 

13 

c 


73 

a 

c3 

■+—* 

c 

<D 

s 

D 

CJ 

c 

a) 

a> 

£ 

a3 

PQ 

' c 
o 

C/3 

c3 

o 

£ 

o 

U 


c 

<u 

JO 


731 


a 


UhI 


cM 


SI 


SI 


o 

4— » 

2" 

'C 

o 

C/3 

<D 

Q 


O 

to 

CO 

X 

OO 

X 

CO 

OO 

X 

o 

o 

CO 

o 

o 

7j- 

o 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

r- 

<N 

] 

On 

oo 

to 

r- 

ON 

CN 

CN 

X 

o 

o 

X 

ON 

o 

r- 

r- 

r^ 

X 

r- 

t- 

X 

X 

to 

ON 

oo 

oo 

r- 

r- 

X 

ON 

ON 

r- 

X 

X 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

* 

* 

* 

* 

7f 

X 

ON 

00 

o 

4— C 

oo 

ON 

o 

Tt 

X 

00 

o 

CO 

CN 

<N 

CN 

CO 

oi 

* 

* 

* 

* 

* 

* 

CN 

r- 

On 

CN 

r- 

o 

r- 

CO 

X 

ON 

X 

Tt 

CN 

X 

o 

CN 

r- 

00 

CO* 

CN 

CN 

r4 

oi 

d 

CN 

ON 

X 

to 

X 

CO 

O 

ON 

to 

X 

CN 

O 

CO 

tJ- 

CN 

to 

CN 

ON 

to 

r- 

to 

ON 

ON 

ON 

r- 

OO 

r- 

00 

oo 

oo 

ON 

On 

On 

oo 

00 

r- 

ON 

ON 

r- 

00 

o* 

d 

d 

d 

d 

d 

d 

d 

o 

o 

o 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

cn 

On 

l 

(N 

to 

c\) 

r- 

X 

to 

O 

CN 

o 

to 

CN 

CN 

o 

CO 

CN 

to 

CN 

On 

o 

On 

CO 

to 

X 

r- 

r- 

ON 

o 

Os 

X 

r- 

7f 

r*** 

X 

7f 

7j" 

d 

d 

d 

d 

d 

d 

d 

o 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

d 

co 

CN 

X 

X 

CN 

OO 

o 

ON 

CN 

tj- 

to 

l 

OO 

7f 

X 

co 

to 

to 

00 

4— C 

X 

X 

to 

<N 

CO 

CM 

X 

X 

X 

00 

ON 

oo 

q 

CN 

r- 

CO 

00 

d 

d 

d 

d 

d 

d 

d 

d 

o 

d 

d 

d 

d 

d 

oi 

oi 

d 

o 

o 

o 

CO 

On 

00 

oo 

oo 

to 

to 

co 

X 

o* 

X 

CO 

CO 

r- 

to 

On 

ON 

ON 

On 

On 

oo 

00 

00 

00 

ON 

ON 

ON 

ON 

On 

OO 

ON 

ON 

ON 

ON 

oo 

o 

O 

d 

d 

O 

O 

o 

o 

d 

d 

o 

d 

d 

O 

d 

d 

d 

d 

d 

d 

d 

ON 

X 

(N 

O 

(N 

r- 

On 

<N 

o\ 

o 

co 

oo 

r- 

7f 

ON 

OO 

CN 

to 

o 

CN 

°o 

CO 

4— H 

o 

00 

r- 

(N 

o 

00 

ON 

o 

to 

CO 

X 

CN 

CN 

to 

o 

r- 

d 

o 

o 

d 

d 

d 

d 

CO 

O 

7f 

to 

r- 

oo 

oo 

ON 

X 

ON 

oo 

ON 

to 

CO 

ON 

X 

ON 

oo 

7j- 

<N 

r-4 

X 

*— < 

r- 

X 

X 

<N 

*— • 

00 

X 

CN 

On 

CN 

Os 

o 

to 

Os 

CO 

CN 

o 

o 

(N 

— 3 

o 

o 

o 

— 3 

d 

d 

CN 

oi 

oi 

d 

oi 

— 1 

■2,  ^ 

5 2 

o '5 

*-»  QJ 

c *3 

rO  on 

PP  a) 

C/3 
. c3 

o a) 

VO  & 


fc* 

D 

S' 


D 


& 

D 


D 


D 

is1 


I § 

w £ 

a,  jg 
x nS 
PP  J 


cd  X 

r-  r- 


c 

<u 

4—4 

X 

PP 

d 

r- 


2 

D 

on 

O 

o, 

X 

pp 


-a 

o 

o 

X 

13 

J* 


D 

3* 


c 

<u 

■4—4 

X 

pp 


5?  oo  oo 

DC  ~ ^ ~ 

00 


c 

.2 

’■4—4 

-3 

c 

o 

U 

C/3 

a 

o 

73 

c3 


a 

3 

on 

O 

CX 

X 

PP 


& 

D 


C 

<D 

•4—4 

X 

pp 


e 

<D 

£ 

CXca 

3 2 
cr  d 
PP  & 


CS  X>  CJ 
On  On  On 


C/3 

D 

O 

73 

c3 


o 

cx 

X 

pp 


DC 

on 


c3  X 
O O 
CN  CN 


3 C 

S’-S 
o I 

c CO 

^ on 

x 2 

w -§ 

cj  ^ 

o a 

CN  HH 


C3 

a) 


& 

►S  & 

4-t  3 

5 'M 

O 4-t 
3 O O 

2 x 4—» 
§ 35  C 
X <L>  <U 
CX  iT>  4—4 

XXX 

PP  d PP 

c3  X cj 
(N  (N  (N 


OX) 

c 


OX) 


o X) 

OX) 

cj  zr 

4JXI  C 

a)  ox)  x 

- c 

on  -3 
& C 

1i*u 
JgS 
SP  hn  M 


J3 

cj 

D 

O 


o 

CN 


CN 


on 

O 

cx 

>> 

O 

.« 

CN 

CN 


a 

OJO 

a 

a 

3 

i— i 

X 

C 

'S2 

.£ 

(D 

’4—* 

c 

c3 

B 

13 

a) 

3 

co 

4—4 

co 

U 

& 

d 

d 

d 

d 

a> 

CN 

CN 

CN 

CN 

CN 

CN 

CN 

CN 

CN 

01 

VO 

CD 

J> 


<rF 

CD 

£> 


O 

ERIC 


73 

<U 

3 

a 

'-4—* 

g 

o 

o 


3 

3 

H 


73 

o 

3 

06 


& 

<u 

3 

E 

•4— > 

on 

U) 

> 


x> 

2 

TS 

73 

G 

3 

Cfl 

G 
O 
• ^ 

3 

a 

G7 

o 

o 


cj 

< 

c/3 

CJ 


cd 

-4— > 

CO 

<D 

> 


Uh 

CJ 

C/3 

<u 

Q 

C/3 

"cd 

G 

< 

73 

G 

cd 

G 

<U 

X) 

E 

3 

CJ 

£ 

G 

<D 

<D 

-4— » 

(U 

PQ 

G 

O 

C/3 

§ 

Di 

E 

o 

U 


C/3 

Id 

G 

< 


X 

<u 

■4—4 

G 

O 

u 

J-H 

o 

£ 

C/3 

G 

.2 

’-4—4 

3 

Dl 

3 

CJ 

CJ 


731 


XI 


PUI 


*i1 


si 


dl 


o 

-4—4 

’C 

CJ 

C/3 

<D 

Q 


Ov 

7f 

o 

r- 

00 

CN 

m 

o 

o 

CO 

<N 

CO 

d 

d 

d 

d 

d 

d 

d 

r^* 

7f 

CN 

VO 

m 

00 

CN 

r- 

(N 

VO 

m 

VO 

Ov 

7f 

vo 

VO 

oo 

00 

VO 

m 

vo 

m 

in 

CO 

vo 

d 

d 

d 

d 

d 

d 

d 

d 

o 

d 

d 

d 

d 

d 

d 

* 

* 

* 

y— I 

CN 

CN 

m 

m 

Ov 

7f 

m 

p 

p 

vq 

O 

CO 

oq 

CN 

CN 

CN 

CN 

* 

* 

* 

* 

* 

* 

r-* 

Ov 

00 

O 

7f 

vo 

m 

r- 

o 

OV 

<N 

CN 

00 

cq 

m 

O) 

d 

7t 

CN 

vd 

CN 

VO 

r- 

m 

VO 

r- 

r- 

VO 

CO 

oo 

r- 

i 

CO 

CN 

1 

m 

r- 

VO 

vo 

Ov 

Ov 

00 

00 

r- 

r- 

r- 

r- 

VO 

00 

d 

d 

d 

d 

d 

d 

d 

o 

o 

o 

d 

o 

d 

d 

d 

CN 

CO 

m 

CN 

vo 

m 

o 

CO 

CN 

vo 

CN 

CO 

VO 

Tf- 

CN 

CN 

7f 

vo 

00 

r- 

m 

r-* 

m 

r-* 

d 

d 

d 

d 

,“H 

o 

o 

o 

o 

o 

d 

o 

d 

vo 

CO 

o 

o 

Ov 

vo 

o 

7f 

CO 

Ov 

7f 

o 

7f 

vo 

vo 

7f 

CN 

vq 

<N 

CO 

OO 

p 

p 

in 

p 

O 

7t 

o 

CN 

’ 1 

d 

CO 

CO 

CO 

CO 

CO 

CN 

r- 

VO 

O 

00 

CN 

in 

ov 

00 

oo 

CN 

CN 

O 

CO 

CN 

oo 

r- 

On 

r- 

Ov 

Ov 

00 

oo 

vo 

r- 

VO 

r- 

r-* 

d 

o 

O 

d 

O 

o 

o 

d 

o 

d 

d 

o 

o 

o 

d 

CN 

o 

Ov 

t 

r- 

00 

vo 

CN 

vo 

o 

CO 

r- 

m 

CN 

00 

VO 

p 

Ov 

Ov 

7f 

tT 

OV 

CO 

OV 

ov 

7f 

CO 

00 

00 

ov 

d 

d 

d 

CN 

o 

d 

d 

d 

d 

d 

d 

d 

vo 

r- 

O 

o 

00 

r- 

CO 

o 

CN 

(N 

m 

r- 

in 

p 

CN 

p 

vo 

in 

CN 

7f 

in 

p 

d 

<N 

<N 

CN 

d 

CO 

CO 

CO 

7f 

CN 

CN 

<D  3 iS 

M ffi  m S 


<x  w>  x:  .x 

CN  M CN  M 
(N  (N  (N  CN 


^ (N  CO  Tf  On 
co  co  co  co  co 


s 


00 


CO 

.iV 


* § g>; 


v s M 

Oio  g 
* .tS  X5 


o 

ERIC 


CD 

J> 


Table  7-16 

Principal  Components  Analysis  Pattern  Matrix  for  the  Analyst  Data:  Work  Context 


Factors 


Descriptor 

FI 

F2 

F3 

F4 

F5 

F6  F7 

Communalitv 

3. 

Communication  Subjectivity 

.75 

-.01 

-.07 

.23 

-.03 

.06  -.52 

.90 

4. 

Social  Interaction 

, .21 

-.08 

.00 

.73 

-.30 

.04  -.44 

.87 

6a. 

Supervise/Develop  Others 

.94 

-.11 

.02 

.08 

-.10 

.05  .01 

.91 

6b. 

Persuade  or  Influence 

.38 

-.01 

-.19 

.58 

.08 

.17  -.41 

.85 

6c. 

Provide  Service  to  Others 

-.19 

-.12 

-.09 

.88 

-.15 

.17  -.12 

.90 

6d. 

Take  Opposing  Position 

.90 

-.03 

-.00 

.17 

-.04 

-.15  -.19 

.89 

6f. 

Deal  With  Public 

.01 

-.14 

-.05 

.93 

-.06 

-.03  -.03 

.93 

6g- 

Coordinate/Lead  Activity 

.87 

-.03 

.06 

-.06 

-.13 

-.01  -.36 

.92 

7. 

Health/Safety  of  Others 

.34 

.44 

.58 

.05 

.03 

.18  -.05 

.87 

8. 

Responsible  Others'  Work 

.95 

-.15 

.07 

-.02 

-.08 

-.04  -.07 

.90 

9. 

Conflict  Situations 

.56 

.08 

-.03 

.65 

-.02 

-.24  -.29 

.92 

10. 

Unpleasant  Individuals 

.08 

.00 

-.01 

.89 

-.13 

-.08  -.18 

.96 

11. 

Physical  Aggression 

.25 

.28 

.43 

.57 

.13 

-.05  -.16 

.92 

12a. 

Indoors,  Controlled 

.03 

-.93 

-.04 

-.01 

-.16 

.13  .08 

.92 

12c. 

Outdoors,  Exposed 

.04 

.91 

-.06 

.12 

.27 

.05  -.19 

.95 

15a. 

Distracting  Noise 

.04 

.74 

.11 

-.25 

.32 

.18  .40 

.92 

15b. 

Extreme  Temperature 

.01 

.79 

-.02 

-.10 

.38 

.29  .01 

.90 

15c. 

Poor  Lighting 

-.02 

.82 

.18 

-.09 

.35 

.09  .09 

.87 

15d. 

Contaminants 

-.03 

.40 

.57 

-.40 

.30 

.30  .21 

.88 

15e. 

Cramped  Work  Space 

-.10 

.44 

.24 

-.16 

.67 

.26  .28 

.88 

15f. 

Whole  Body  Vibration 

.13 

.82 

.10 

-.26 

.10 

-.08  .22 

.91 

16a. 

Radiation 

-.08 

.13 

.90 

-.01 

-.05 

.11  -.09 

.86 

17a. 

Diseases 

.09 

-.20 

.85 

.06 

-.12 

.26  -.10 

.91 

18a. 

High  Places 

.05 

.27 

.11 

-.07 

.90 

.12  .02 

.92 

19a. 

Hazardous  Conditions 

.13 

.14 

.57 

-.48 

.45 

.23  .10 

.86 

20a. 

Hazardous  Equipment 

.00 

.47 

.17 

-.39 

.51 

.28  .41 

.92 

21a. 

Hazardous  Situations 

.01 

.23 

.31 

-.28 

.22 

.67  .29 

.86 

22a. 

Sitting 

.15 

-.07 

-.17 

.08 

-.25 

-.90  -.04 

.93 

22b. 

Standing 

-.11 

.03 

.16 

.03 

.25 

.91  .05 

.95 

22c. 

Climbing  Ladders,  etc. 

-.09 

.25 

-.06 

-.18 

.84 

.28  -.00 

.91 

22d. 

Walking  or  Running 

.06 

-.08 

.16 

.29 

.36 

.70  -.32 

.84 

22e. 

Kneeling  or  Crouching 

-.20 

.36 

.11 

-.11 

.62 

.45  .00 

.80 

22f. 

Keeping/Regaining  Balance 

.01 

.28 

.08 

-.03 

.88 

.15  .05 

.90 

22g. 

Handling  Tools,  Objects 

-.25 

.21 

.32 

-.34 

.29 

.23  .60 

.82 

22h. 

Bending/Twisting  Body 

-.19 

.45 

.28 

-.10 

.59 

.45  .22 

.93 

22i. 

Making  Repetitive  Motions 

-.45 

-.02 

.00 

-.19 

.00 

.09  .80 

.88 

23b. 

Special  Uniform 

.03 

.06 

.70 

.32 

.15 

.45  .22 

.91 

23d. 

Common  Safety  Attire 

.08 

.10 

.75 

-.34 

.35 

.30  .17 

.93 

23e. 

Special  Safety  Attire 

.17 

.10 

.84 

-.21 

.38 

.08  .04 

.90 

24. 

Consequence  of  Error 

.52 

.15 

.67 

.10 

.12 

-.30  .20 

.92 
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Table  7-16  (continued) 

Principal  Components  Analysis  Pattern  Matrix  for  the  Analyst  Data;  Work  Context 


Factors 


DescriDtor 

FI 

F2 

F3 

F4 

F5 

F6 

F7 

28. 

Frustrating  Circumstances 

.81 

.11 

.22 

-.00 

.21 

-.21 

.06 

83 

29. 

Level  of  Automation 

.27 

-.17 

.27 

-.34 

-.18 

-.45 

.50 

89 

31, 

Accuracy/Exactness 

.28 

-.48 

.51 

-.14 

-.12 

-.41 

.19 

.07 

85 

32. 

Details  and  Completeness 

.44 

-.42 

.57 

-.10 

-.03 

-.31 

.12 

83 

33. 

Constant  Awareness 

.54 

.41 

.44 

.31 

.06 

-.02 

-.05 

.0*7 

85 

34. 

Repetitive  Activities 

-.42 

-.20 

-.22 

-.19 

.22 

-.10 

.68 

83 

39. 

Machine  Driven  Pace 

-.10 

.32 

.14 

-.52 

.06 

.01 

.70 

.91 

Percent  of  Variance 

31 

19 

14 

9 

6 

3 

3 

Eigenvalue 

14.54 

9.01 

6.58 

4.50 

2.99 

1.85 

1.39 

H^te.  N = 37.  The  Correlation  Matrix  was  based  on  means  calculated  at  the  occupation  level. 
TCiese  loadings  are  based  on  an  orthoganal  varimax  rotation.  FI  = Managerial  Relations,  F2  = 
Environmental  Conditions,  F3  = Health  and  Safety  Conditions,  F4  = Interacting  with  Public  F5 
= Physical  Activity,  F6  = Body  Movement,  F7  = Structured/Machine  Operations. 


Table  7- 17a 

Mean  Differences  Between  Exempt  and  Non-Exempt  Jobs:  Work  Context 


Exempt  Non-Exempt 

(n  = 36) (n=  112) 


Descriptor 

M 

SD 

M 

SD 

t 

6.  Job  Interactions 

6a.Supervise/Develop  Others 

3.53 

1.30 

2.90 

1.51 

2.24* 

6b.Persuade  or  Influence 

2.78 

2.03 

1.62 

1.80 

3.24* 

6g.Coordinate/Lead  Activity 

3.42 

1.40 

2.74 

1.60 

2.27* 

8.  Responsible  Others'  Work 

3.92 

2.32 

3.36 

2.40 

1.21 

12.  Work  Settings 

12c.  Outdoors,  Exposed 

0.58 

1.22 

2.29 

2.65 

- 3.74* 

12e.  Open  Vehicle/Equip. 

0.00 

0.00 

0.49 

1.44 

-2.03* 

12f.  Enclosed  Vehicle/Equip. 

0.81 

1.85 

1.72 

2.50 

-2.03* 

15.  Environmental  Conditions 

15a.  Distracting  Noise 

2.92 

2.70 

3.27 

2.54 

-0.72 

15b.  Extreme  Temperature 

1.28 

2.13 

2.08 

2.38 

-1.90* 

15c.  Poor  Lighting 

0.58 

1.48 

1.32 

2.23 

-2.27* 

15d.  Contaminants 

1.64 

2.52 

2.66 

2.52 

-2.12* 

15e.  Cramped  Work  Space 

0.47 

1.25 

1.59 

2.03 

-3.93* 

15f.  Whole  Body  Vibration 

0.14 

0.83 

0.42 

1.39 

-1.47 

16.  Radiation 

16a.  Exposure 

0.03 

0.17 

0.44 

1.36 

-3.18* 

17.  Diseases/Infections 

17a.  Exposure 

0.67 

1.69 

2.48 

2.63 

-4.81* 

18.  High  Places 

18a.  Exposure 

0.39 

1.42 

0.90 

1.74 

-1.78* 

19.  Hazardous  Conditions 

19a.  Exposure 

0.22 

0.54 

1.44 

2.13 

-5.55* 

20.  Hazardous  Equipment 

20a.  Exposure 

0.72 

1.75 

1.89 

2.60 

-3.07* 

21.  Hazardous  Situations 

21a.  Exposure 

0.92 

1.61 

2.62 

2.37 

-4.86* 

22.  Body  Positioning 

22a.  Sitting 

3.03 

1.63 

2.44 

1.49 

2.02* 

22b.  Standing 

2.81 

1.58 

3.11 

1.51 

-1.03 

22c.  Climbing  Ladders,  etc. 

0.25 

0.55 

0.43 

0.80 

-1.24 

22d.  Walking  or  Running 

1.92 

1.63 

2.27 

1.47 

-1.21 

22e.  Kneeling  or  Crouching 

0.67 

1.07 

1.00 

1.21 

-1.48 

22f.  Keeping/Regaining  Balance 

0.47 

1.18 

0.66 

1.11 

-0.86 

22g.  Handling  Tools,  Objects 

1.86 

1.78 

2.74 

1.93 

-2.42* 

22h.  Bending/Twisting  Body 

1.03 

1.52 

1.62 

1.57 

-2.00* 

22i.  Making  Repetitive  Motions 

2.14 

1.93 

2.77 

1.68 

-1.88* 

29.  Level  of  Automation 

2.96 

1.73 

3.22 

1.93 

-0.75 
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Table  7- 17a  (continued) 

Mean  Differences  Between  Exempt  and  Non-Exempt  Jobs:  Work  Context 


Descriptor 

35.  Unstructured  Tasks/Goals 
39.  Machine  Driven  Pace 
*E  < .05 


Exempt  Non-Exempt 

(n  = 36) (n  = 112) 

M SD  M SD  t 

4.83  1.98  4.24  1.44  -L65 

1-35  1.83  1.71  1.72  -1.08 


A total  of  9 occupations  are  classified  as  Exempt  and  28  occupations  are  classified  as  Non- 
Exempt.  Four  individuals  were  randomly  sampled  from  each  occupation  to  be  included  in  these 
analyses. 


Non-Exempt  Occupations 

Educational  Administrators 

General  Managers  & Top  Executives 

Chemical  Engineers 

Mechanical  Engineers 

Teachers,  Preschool 

Teachers,  Elementary  School 

Salespersons,  Except  Scientific  & Retail 

First-Line  Supervisors,  Clerical  & Administrative 

Police  & Detective  Supervisors 

Insurance  Claims  Clerks 

Teachers’  Aides  & Assistants,  Clerical 

Secretaries,  Except  Legal  & Medical 

Receptionists  & Information  Clerks 

Bookkeeping,  Accounting  & Auditing  Clerks 

General  Office  Clerks 

Police  Patrol  Officers 

Waiters  & Waitresses 

Cooks,  Restaurant 

Food  Preparation  Workers 

Medical  Assistants 

Nursing  Aides,  Orderlies  & Attendants 
Janitors  & Cleaners 

Maintenance  Repairers,  General  Utility 
Earth  Drillers,  Except  Oil  & Gas 
Machinists 

Packaging  & Filling  Machine  Operations 
Truck  Drivers,  Heavy  or  Tractor-Trailer 
Bus  Drivers,  Schools 


O 

ERJC 
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Table  7- 17a  (continued) 

Mean  Differences  Between  Exempt  and  Non-Exempt  Jobs:  Work  Context 


Exempt  Occupations 
Computer  Programmers 
Registered  Nurses 

Medical  & Clinical  Laboratory  Technologists 

Medical  & Clinical  Laboratory  Technicians 

Salespersons,  Retail 

Stock  Clerks,  Sales  Floor 

Cashiers 

Tellers 

Loan  & Credit  Clerks 
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Table  7-17b 

Mean  Differences  Between  High  Teamwork  and  Low  Teamwork  Jobs:  Work  Context 


High  Teamwork Low  Teamwork 

(n  = 44) (n  = 52) 


Descriptor 

M 

SD 

M 

SD 

t 

1.  Communication  Formality 

4.52 

1.81 

3.71 

1.55 

2.36* 

2.  Communication  Methods 

2a.  Face-to-Face  Individuals 

5.64 

1.71 

4.77 

2.26 

2.09* 

2b.  Face-to-Face  Groups 

2.43 

1.81 

1.12 

1.00 

4.30* 

2c.  Public  Speaking 

1.37 

2.26 

0.44 

1.38 

2.38* 

2d.  Video  Conference 

0.34 

1.16 

0.04 

0.19 

1.71 

2e.  Voice  Mail 

1.05 

1.57 

1.57 

2.11 

-1.38 

2f.  Telephone 

4.07 

2.06 

4.11 

2.50 

-0.09 

2g.  Interactive  Computer 

1.39 

2.42 

1.54 

2.49 

-0.30 

2h.  Electronic  Mail 

1.14 

1.65 

1.15 

1.97 

-0.05 

2i.  Handwritten  Notes 

3.23 

1.57 

3.38 

1.83 

-0.45 

3.  Communication  Subjectivity 

3.75 

1.73 

3.00 

1.47 

2.31* 

4.  Social  Interaction 

5.95 

1.51 

5.53 

2.00 

1.19 

6.  Job  Interactions 

6a.  Supervise/Develop  Others 

3.56 

1.41 

2.42 

1.45 

3.90* 

6b.  Persuade  or  Influence 

2.00 

2.05 

1.13 

1.44 

2.34* 

6c.  Provide  Service  to  Others 

3.91 

1.68 

2.96 

2.03 

2.50* 

6d.  Take  Opposing  Position 

1.85 

1.66 

1.23 

1.29 

2.00* 

6e.  Work/Contribute  To  Team 

4.41 

0.58 

3.37 

1.43 

4.82* 

6g.  Coordinate/Lead  Activity 

3.07 

1.62 

2.44 

1.65 

1.88* 

8.  Responsible  Others'  Work 

3.57 

2.50 

2.94 

2.43 

1.24 

9.  Conflict  Situations 

1.91 

1.01 

1.62 

0.91 

1.50 

24.  Consequence  of  Error 

4.21 

2.25 

2.62 

1.87 

3.78* 

25.  Impact  of  Decisions 

25a.  Level  of  Decisions 

4.75 

1.53 

3.90 

1.90 

2.37* 

25b.  Frequency  of  Decisions 

3.91 

2.41 

3.25 

2.50 

1.31 

28.  Frustrating  Circumstances 

3.02 

1.87 

3.33 

1.59 

-0.86 

35.  Unstructured  Tasks/Goals 

4.34 

1.83 

4.38 

1.59 

-0.13 

38.  Work  With  Distractions 

3.31 

1.30 

3.15 

1.49 

0.54 

*E  < .05 

Note.  A total  of  1 1 occupations  are  classified  as  High  Teamwork  and  13  occupations  are 
classified  as  Low  Teamwork.  Four  individuals  were  randomly  sampled  from  each  occupation  to 
be  included  in  these  analyses. 


Low  Teamwork 
Educational  Administrators 
Computer  Programmers 
Insurance  Claims  Clerks 
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Table  7- 17b  (continued) 

Mean  Differences  Between  High  Teamwork  and  Low  Teamwork  Jobs:  Work  Context 

General  Office  Clerks 
Police  & Detective  Supervisors 
Medical  Assistants 

Nursing  Aides,  Orderlies  & Attendants 

Janitors  & Cleaners 

Earth  Drillers,  Except  Oil  & Gas 

Packaging  & Filling  Machine  Operations 

Bus  Drivers,  Schools 

Machinists 

Truck  Drivers,  Heavy  or  Tractor-Trailer 

High  Teamwork 

Mechanical  Engineers 

Salespersons,  Except  Scientific  & Retail 

Salespersons,  Retail 

Stock  Clerks,  Sales  Floor 

Cashiers 

First-Line  Supervisors,  Clerical  & Administrative 
Receptionists  & Information  Clerks 
Bookkeeping,  Accounting  & Auditing  Clerks 
Police  Patrol  Officers 
Waiters  & Waitresses 
Maintenance  Repairers,  General  Utility 
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Chapter  8 

Organizational  Context: 

Evidence  for  the  Reliability  and  Validity  of  the  Measures 

Sharon  Arad 
Mary  Ann  Hanson 
Robert  J.  Schneider 

Personnel  Decisions  Research  Institutes,  Incorporated 


Background 

The  inclusion  of  a set  of  organizational  context  descriptors  in  the  0*NET  content  model 
was  driven  by  the  growing  demand  for  descriptive  information  about  organizations  in  the  United 
States  on  the  part  of  many  Dictionary  of  Occupation  Titles  (DOT;  U.S.  Department  of  Labor, 
1991)  users.  It  is  reasonable  to  expect  the  organizations  that  provide  the  context  within  which 
jobs  occur  to  affect  the  very  nature  of  the  jobs  themselves.  Thus,  descriptive  information  about 
organizations  is  likely  to  aid  in  classifying  and  clustering  occupations.  Concern  about  the 
organizational  and  administrative  effectiveness  of  American  business  has  led  to  the  identification 
of  a set  of  business  practices  typical  of  organizations  competing  effectively  in  the  national  and 
global  economies  (U.S.  DOL,  1994).  This  concept  of  “high-performance”  business  practices  and 
workplaces  is  a new  one,  and  only  a limited  amount  of  information  is  available  concerning  the 
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extent  to  which  businesses  actually  use  these  “high-performance”  practices,  as  well  as  the  extent 
to  which  these  practices  are  related  to  organizational  effectiveness  when  they  are  used.  Thus,  the 
description  of  high-performance  business  practices  was  a high  priority  in  developing  the  content 
model  underlying  the  0*NET. 

Taxonomy 

The  0*NET  measures  of  organizational  context  characteristics  were  derived  from  the 
taxonomy  of  organizational  context  proposed  by  Arad,  Schneider,  and  Hanson  (1995).  This 
taxoriomy  was  based  on  an  extensive  review  and  integration  of  major  theoretical  and  empirical 
writings  on  organizations.  Constructs  were  selected  for  the  taxonomy  based  on  the  following 
criteria:  ( 1 ) they  had  been  measured  with  reasonable  levels  of  reliability  and  validity  in  past 
research,  or  had  good  potential  for  being  reliable  and  valid,  (2)  they  could  be  generalized  to  a 
wide  variety  of  different  types  of  organizations  without  losing  their  meaning,  and  (3)  they  were 
expected,  based  on  theory  or  past  research,  to  be  useful  in  describing  or  classifying  occupations, 
or  in  describing  important  features  of  organizations  that  might  be  of  interest  to  0*NET  users.  In 
addition  to  these  criteria,  objectivity  of  descriptors  was  considered  important.  The  simplest 
approach  to  collecting  organizational  data  is  to  survey  a single  organizational  representative,  but 
this  makes  it  impossible  to  even  assess  the  reliability  of  the  resulting  data.  However,  reliability 
can  be  viewed  as  less  of  a concern  for  questions  that  ask  for  relatively  objective  information 
(e.g.,  numbers  of  employees,  types  of  forms  used);  thus,  an  additional  goal  was  to  keep  the 
descriptors  as  objective  and  concrete  as  possible. 

Based  on  these  criteria,  Arad,  Schneider,  and  Hanson  (1995)  identified  a variety  of 
relatively  specific  organizational  context  constructs  that  appeared  useful,  and  organized  these 
constructs  into  a hierarchical  taxonomy.  These  constructs  were  grouped  according  to  six  higher- 
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order  topic  areas,  which  provide  a useful  heuristic  for  categorizing  the  lower-order  constructs: 
type  of  industry,  organizational  structure,  human  resources  fHR)  systems  and  practices,  culture. 
goals,  and  roles.  Organizational  structure  and  HR  systems  and  practices  are  both  structural 
characteristics,  while  culture,  goals,  and  roles  are  social  processes. 

Type  of  industry  was  included  in  the  organizational  context  taxonomy  since  it  was  part  of 
the  DOT  and  many  users  have  indicated  interest  in  this  kind  of  information.  Organizational 
structure  constitutes  a major  element  in  the  taxonomy.  Structure  plays  a central  role  in  virtually 
every  organizational  theory,  and  is  generally  viewed  as  affecting  the  behavior  of  job  incumbents 
as  well  as  the  ability  of  organizations  to  adapt  effectively  to  their  environments.  Seven  important 
structural  elements  of  organizations  were  included  in  the  0*NET  taxonomy:  organization  and 
establishment  size,  hierarchy,  specialization,  formalization,  standardization,  centralization  and 
employee  empowerment,  specialization,  and  job  characteristics  (skill  variety,  task  significance- 
task  identity,  autonomy,  and  feedback).  Four  additional  structural  constructs,  often  associated 
with  high-performance  organizations,  were  also  included  in  the  taxonomy:  information  sharing, 
team  structure,  type  of  work  teams,  and  amount  of  change  in  the  organization  structure. 
Information  sharing  and  amount  of  change  in  the  organization  structure  were  not  included  in  the 
original  organizational  context  taxonomy  described  by  Arad,  Schneider,  and  Hanson  (1995),  but 
were  added  based  on  subsequent  discussions  with  organizational  representatives.  Two  constructs 
that  were  in  the  original  taxonomy  were  dropped:  administrative  intensity  and  span  of  control. 

Arad,  Schneider,  and  Hanson  (1995)  proposed  that  a comprehensive  description  of  the 
human  resources  (HR)  systems  and  practices  of  organizations  needs  to  include  various  HR 
domains,  such  as  organizational  socialization  practices,  organizational  reward  systems, 
recruitment  and  selection  practices,  and  employee  training  and  development.  Accordingly,  the 

777 
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following  HR  constructs  were  included  in  the  0*NET  taxonomy:  recruitment  planning  anH 
operations,  selection  processes  and  methods  used,  group  and  individualized  socialization. 
training  topics/programs  and  methods  used,  extent/support  of  training,  use  of  data  in  training 
process,  basis  for  compensation,  and  benefits  and  compensation  elements.  Two  additional 
constructs  thought  to  be  characteristic  of  high-performance  organizations  were  also  added  to  this 
domain:  use  of  independent  contractors,  and  use  of  data  in  organizational  decision-making  in 
general.  The  construct  called  all  salaried  system  was  not  included  as  a separate  category  of 
variables. 

/ 

Organizational  culture  was  assessed  by  including  measures  of  organizational  values, 
while  two  distinct  goal  setting  literatures  were  the  basis  for  the  development  of  goal  setting 
descriptors:  individual  goal  setting  and  organizational  goal  setting.  For  both  the  individual  and 
organizational  levels,  three  goal  constructs  were  included:  goal  specificity,  availability  of  goal 
feedback,  and  goal  negotiability.  These  have  been  refined  somewhat  from  the  set  of  goal 
constructs  in  the  earlier  report  (extent  of  individual  goal  setting,  individual  goal  characteristics, 
availability  of  goal  feedback,  method  of  goal  assignment,  extent  of  organizational  goal  setting, 
and  organizational  goal  characteristics'). 

Another  organizational  context  construct  likely  to  affect  job  requirements  is  roles.  Role 
constructs  included  in  the  0*NET  taxonomy  are:  role  conflict,  role  overload,  and  role 
negotiability.  Measures  of  four  leader  behaviors  or  styles  were  also  included  in  the  0*NET: 
consideration,  task-orientation,  visionary,  and  problem  solving  leadership  (these  were  originally 
included  in  the  work  context  domain).  A measure  of  changes  in  leadership  (i.e.,  changes  in 
supervision)  was  also  included  in  the  taxonomy.  Organizational  culture  was  assessed  by 
including  measures  of  organizational  values. 


Organizational  Context  8-5 

Samples  and  Measures 

The  general  approach  taken  in  measuring  organizational  context  characteristics  was  to 
obtain  as  much  of  the  information  as  possible  from  a single  organizational  representative.  This 
was  done  for  two  reasons.  First,  incumbent  time  was  at  a premium  in  the  0*NET  data  collection. 
Virtually  all  of  the  other  areas  in  the  content  model  rely  heavily  on  incumbent  data  and  require  a 
fair  amount  of  incumbent  time.  Secondly,  incumbent  data  concerning  organizational  descriptors 
is  confounded,  because  incumbents  necessarily  answer  from  the  perspective  of  their  job  or 
occupation.  Unless  we  are  able  to  sample  a large  number  of  representative  occupations  from  a 
single  organization,  it  will  be  impossible  to  even  assess  the  extent  to  which  this  confound  affects 
the  data. 

Still,  some  of  the  organizational  context  constructs  could  not  be  measured  appropriately 
at  the  organizational  level  and  were  therefore  included  in  the  incumbent  questionnaire. 

Constructs  reflecting  elements  of  organizational  structure,  HR  practices,  and  roles  were  therefore 
included  in  the  incumbent  questionnaire.  For  example,  job  characteristics  have  traditionally  been 
measured  by  surveying  individual  incumbents,  so  scales  targeting  each  of  the  five  job 
characteristics  associated  with  job  enrichment  (i.e.,  autonomy,  skill  variety,  task  identity,  task 
significance,  and  feedback)  were  included  in  the  incumbent  questionnaire.  Similarly,  questions 
concerning  employee  empowerment,  role  conflict,  and  role  overload  seemed  more  appropriately 
asked  of  job  incumbents  and  were  included  in  the  incumbent  questionnaire.  Leader 
behavior/style  variables  were  also  deemed  more  appropriate  at  the  incumbent  level  for  the 
present  research.  Leader  behavior/style  was  measured  by  four  items,  written  by  the  authors, 
which  were  designed  to  assess  four  different  dimensions  of  leader  behavior/style:  consideration, 
task  orientation,  visioning,  and  problem  solving.  Incumbents  were  asked  to  rate  the  frequency 
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with  which  their  immediate  supervisor  demonstrated  behaviors  related  to  each  of  these 
dimensions.  Information  concerning  recruitment  sources  and  selection  methods  used  is  likely  to 
be  of  most  interest  to  0*NET  users  as  it  related  to  particular  occupations,  so  data  concerning 
these  descriptors  were  collected  from  incumbents  only  as  well.  Finally,  extent  of  technical 
training  was  measured  by  asking  incumbents  how  often  they  receive  job-related  training. 

Other  organizational  context  constructs,  where  both  the  organizational  and  incumbent 

perspectives  were  expected  to  provide  different  but  relevant  information  (e.g.,  goal  setting, 

recruitment  practices,  and  centralization),  were  included  in  the  job  incumbent  questionnaire,  as 

well  as  the  organizational  representative  interview.  These  constructs  comprised  a variety  of 

organizational  structure,  HR  practices,  and  culture  variables.  We  measured  the  following 

» 

variables  at  both  the  organization  and  the  individual  level:  centralization,  training 
topics/programs  and  methods  used,  job  rotation  policies,  extent  of  use  of  teams,  benefits  and 
compensation  elements,  and  organizational  values.  Questions  concerning  these  variables  in  the 
incumbent  questionnaire  asked  respondents  to  focus  on  their  specific  jobs.  Questions  concerning 
these  variables  m the  organizational  representative  survey  asked  about  the  organization  as  a 
whole.  The  organizational  values  questions  in  the  incumbent  survey  were  a subset  of  those 
included  in  the  organizational  representative  survey.  These  items  were  reworded  in  order  to 
ensure  that  they  would  be  understandable  to  a wide  variety  of  incumbents.  Figure  8-1  lists  the 
elements  in  the  revised  taxonomy  and  the  items  in  the  job  incumbent  and  organizational 
representative  surveys  that  measure  them  (a  few  items  from  the  questionnaires  were  general 
background  questions  and  as  such  do  not  fit  into  the  taxonomy  and  are  not  included  in  this  figure 
or  in  the  analyses).  Figure  8-2  assigns  numbers  and  labels  to  and  describes  the  33  organizational 
context  descriptors  and  scales  used  in  the  analyses  of  the  incumbent  questionnaire  data.  These 
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descriptor  labels  and  numbers  will  be  used  throughout  the  chapter  in  discussing  the  incumbent 
data. 

Data  concerning  the  remaining  organizational  context  constructs  in  the  content  model 
were  collected  from  the  organizational  representative  only.  In  each  sampled  organization,  we 
asked  for  a single  organizational  representative  who  was  either  a representative  of  the  human 
resources  function  or  someone  from  management.  Data  were  collected  from  these  individuals 
using  a computer  assisted  telephone  interviews  (CATI).  The  CATI  includes  70  questions 
measuring  constructs  from  four  of  the  organizational  context  content  domains:  organizational 
structure,  HR  practices,  goal  setting,  and  culture.  The  remaining  topic  area,  type  of  industry,  was 
not  collected  from  the  organizational  representatives  since  it  could  be  derived  from  the  SIC  codes 
(SIC  manual,  1987).  In  general,  the  CATI  interview  items  ask  respondents  to  rate  the  extent, 
frequency,  or  existence  of  each  of  the  organizational  characteristics,  using  Likert-type  rating 
scales,  check  lists,  or  simple  yes/no  questions. 

The  CATIs,  each  lasting  approximately  half  an  hour,  were  conducted  by  professional 
interviewers  from  Westat,  Incorporated.  The  interview  protocol  is  provided  in  Appendix  A.  This 
protocol  shows  the  exact  content  and  wording  of  each  of  the  organizational  representative 
interviews.  The  computer-assisted  nature  of  the  CATI  allowed  the  interviewers  to  skip  irrelevant 
or  unnecessary  questions,  based  on  the  interviewees  responses  to  previous  items.  It  also 
systematically  prompted  interviewers  to  ask  follow-up  questions  or  provide  clarification  when 
interviewees  had  trouble  with  particular  questions. 

Seven  hundred  and  thirty-three  respondents,  representing  53  different  occupations, 
completed  the  job  incumbent  survey.  The  sampled  occupations  ranged  from  general  managers 
and  top  executives  to  janitors.  The  incumbents  also  represented  a broad  and  diverse  cross-section 
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of  the  workforce  in  terms  of  age,  sex,  ethnic  status,  and  education.  The  sample  included  both  job 
holders  (71%)  and  supervisors  (29%).  Six  hundred  and  sixty-one  organizational  representatives 
were  interviewed;  the  majority  worked  in  personnel  or  human  resources  (61%)  and  virtually  all 
of  the  others  were  managers  or  representatives  of  higher  management.  The  661  sampled 
establishments  represented  almost  every  industry  category  as  well  as  private,  government,  profit, 
and  non-profit  organizations.  The  size  of  these  establishments  ranged  from  5 to  6,000  employees 
(including  both  full  and  part-time).  Seventy  of  the  sampled  establishments  were  also  represented 
in  the  incumbent  sample.  The  remaining  establishments  either  refused  to  participate  in  incumbent 
surveying  after  completing  the  CATI  survey  or  did  not  return  any  completed  incumbent  surveys. 
The  entire  CATI  sample  was  used  for  many  of  the  analyses  presented  here,  but  only  the  70  for 
which  incumbent  data  were  available  could  be  used  in  exploring  relationships  between 
incumbent  and  organizational  representative  data. 

Overview  of  Organizational  Context  Analyses 

Analyses  for  the  organizational  context  descriptors  are  necessarily  somewhat  different 
from  those  for  the  other  content  model  domains,  in  part  because  data  were  collected  from  both 
incumbents  and  organizational  representatives  and  in  part  because  the  focus  is  on  describing 
organizations  in  addition  to  occupations.  The  analyses  of  the  organizational  context  Hata  fall  into 
three  general  categories:  analyses  of  the  incumbent  data,  analyses  of  the  organizational 
representative  (i.e.,  CATI)  data,  and  analyses  aimed  at  understanding  the  commonalities  and 
differences  between  these  two  sources  of  data. 

For  both  the  incumbent  and  the  organizational  representative  data  analyses,  descriptive 
statistics  and  reliability  estimates  were  computed.  For  the  incumbent  data,  this  included 
estimating  interrater  reliabilities  and  the  variance  due  to  the  descriptors,  occupations. 
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organizations  and  the  relevant  interactions.  In  both  incumbent  and  organizational  representative 
descriptors,  data  sets  were  intercorrelated  and  factor  analyzed  to  assess  the  underlying  structure, 
including  both  exploratory  and  confirmatory  analyses.  Comparisons  of  occupational  means  in  the 
incumbent  data  and  discriminant  analyses  were  employed  to  assess  the  potential  usefulness  of 
these  descriptors  in  describing  and  classifying  occupations.  For  the  organizational  representative 
data,  comparisons  of  mean  scores  obtained  by  organizations  representing  different  types  of 
industries  were  conducted  to  provide  a preliminary  assessment  of  the  usefulness  of  these 
descriptors  for  describing  and  classifying  organizations.  Finally,  comparisons  between  the 
incumbent  and  organizational  representative  data  sets  involved  examining  the  underlying 
structure  (i.e.,  factor  analyses)  and  the  correlations  between  conceptually  related  descriptors. 

Each  of  these  analyses  is  described  in  more  detail  in  the  following  section. 

Results 

Incumbent  Data  Analysis  Results 

It  is  important  to  keep  in  mind  that  even  the  incumbent-level  descriptors  in  this  domain 
were  developed  for  the  purpose  of  describing  organizations,  whereas  descriptors  in  other 
domains  were  developed  for  the  purpose  of  describing  jobs  and  occupations.  Data  were  collected 
from  job  incumbents  only  concerning  those  descriptors  for  which  it  was  determined  that  the 
incumbent  perspective  was  the  most  meaningful  (e.g..  Empowerment  [Descriptor  # 1])  and  those 
descriptors  for  which  data  itself  is  interesting  at  the  job  or  occupation  level  (e.g..  Number  of 
Selection  Methods  [Descriptor  #29],  Number  of  Compensation  Elements  [Descriptor  #31]).  Data 
collected  from  job  incumbents  concerning  these  organizational  context  descriptors  can  be  viewed 
as  containing  variance  due  to  the  organizations  in  which  they  work,  but  also  variance  due  to  the 
way  in  which  these  organizations  impact  individual  job  holders.  It  is  also  likely  that,  for  at  least 
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some  of  the  descriptors,  a portion  of  the  variance  is  directly  due  the  respondents’  occupations. 
For  example,  one  would  expect  managers  to,  on  average,  report  more  Empowerment  (Descriptor 
#1)  than  janitors,  even  if  there  are  also  systematic  differences  in  the  overall  degree  of  employee 
empowerment  across  organizations.  Thus,  these  data  were  examined  from  three  perspectives:  (1) 
their  usefulness  in  describing  organizations,  (2)  their  usefulness  in  describing  occupations,  and 
(3)  their  usefulness  in  describing  occupation  within  organization  “cells,”  that  is,  incumbents 
working  in  the  same  occupation  and  the  same  organizations. 

Data  Cleaning 

In  order  to  compute  meaningful  descriptive  statistics  and  reliabilities  across  occupations 
and  across  organizations,  we  included  occupations  and  organizations  in  our  sample  only  if 
responses  were  available  from  four  or  more  incumbents.  As  a result,  the  sample  used  in  the 
analyses  reported  here  included  554  incumbents  who  represented  30  occupations  and  70 
establishments.  Figure  8-3  lists  these  occupations  and  the  numbers  of  incumbents  in  each 
occupation  who  completed  the  organizational  context  questionnaire. 

Descriptive  Statistics 

We  computed  means  and  standard  deviations  once  across  occupations  (using 
occupational  means  as  the  unit  of  analysis)  and  once  across  organizations  (using  organizational 
means  as  the  unit  of  analysis),  and  these  descriptive  statistics  are  presented  in  Table  8-1.  For  the 
a priori  scales  (i.e.,  scales  adapted  from  the  literature).  Table  8-1  provides  only  scale-level 
statistics.  Brief  descriptions  of  the  rating  scales  used  to  collect  information  concerning  each 
scale/descriptor  are  also  provided  in  Table  8-1.  Not  surprisingly.  Table  8-1  shows  very  similar 
means  and  standard  deviations  across  occupations  and  across  organisations 
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Reliability 

For  each  of  the  a priori  scales  (i.e.,  scales  adapted  from  the  literature),  we  computed  an 
estimate  of  internal  consistency  reliability  (coefficient  alpha),  and  these  estimates  are  also 
presented  on  Table  8-1.  For  organizational  values,  the  a priori  scales  were  based  on  the  factor 
analysis  of  the  organizational  representative  data  (described  later),  which  partially  replicated  past 
research  (O’Reilly,  Chatman,  & Caldwell,  1991).  For  most  scales,  the  internal  consistency 
reliability  estimates  were  above  .70.  The  lowest  internal  consistency  estimates  were  for  the  two- 
item  scales:  Role  Negotiability  (Descriptor  #10),  Goal  Specificity  (Descriptor  #1 1),  Attention  to 
Detail  Values  (Descriptor  #16),  and  Role  Overload  (Descriptor  #8)  (.31,  .44,  .45,  and  .61, 
respectively). 

Interrater  reliability  for  these  descriptors  can  be  viewed  in  three  ways:  (1)  agreement 
between  incumbents  who  are  in  the  same  occupation,  (2)  agreement  between  incumbents  who  are 
in  the  same  organization,  regardless  of  their  occupation,  (3)  and  agreement  between  incumbents 
who  are  in  the  same  occupation  and  the  same  organization.  For  descriptors  that  vary 
systematically  across  both  occupations  and  organizations  the  latter  type  of  reliability,  across 
occupation  within  organization  “cells,”  was  expected  to  be  the  highest.  Table  8-1  presents 
interrater  reliabilities  computed  across  occupations  (based  on  a harmonic  mean  of  10.14  judges 
per  occupation)  and  across  organizations  (based  on  a harmonic  mean  of  6.51  judges  per 
organization),  along  with  the  associated  standard  errors  of  measurement.  In  general,  reliabilities 
across  occupations  were  higher  than  those  across  organizations,  but  the  number  of  raters  per 
occupation  is  higher  than  the  number  of  raters  per  organization. 

Table  8-2  shows  estimates  of  the  interrater  reliability  that  would  have  been  obtained  if  30 
judges  had  been  available  for  each  occupation  and  also  the  reliability  of  ratings  provided  by  a 
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single  rater.  This  table  also  shows  the  one-rater  and  30-rater  reliability  estimates  across 
organizations,  and  across  the  occupation  within  organization  cells.  Table  8-2  shows  that,  in 
general,  reliabilities  across  occupations  are  higher  than  those  across  organizations,  even  when  the 
number  of  raters  is  the  same.  On  the  average  however,  reliabilities  across  the  occupation  within 
organization  cells  were  the  highest.  Table  8-2  shows  that  for  several  variables-Task  Identity 
(Descriptor  #3),  Feedback  (Descriptor  #6),  the  four  organizational  value  scales  (Descriptors  #14, 
#15,  #16,  and  #17),  and  two  of  the  leader  styles:  Consideration  (Descriptor  #18)  and  Visionary 
(Descriptor  #20)~these  estimates  are  all  quite  low  (all  across  occupation  jj  estimates  < .05). 
These  descriptors  would  have  low  reliabilities  even  if  30  judges  were  available  for  each 
occupation.  Consequently,  these  descriptors  were  not  included  in  subsequent  advanced  analyses. 
They  were  however  included  in  other  analyses  designed  to  further  assess  the  reliability  of  the 
descriptors  (i.e.,  the  ANOVAs  and  assessment  of  occupation  differences). 

A different  way  to  examine  the  interrater  reliability  of  the  organizational  context 
descriptors  is  to  compare  patterns  or  profiles  of  organizational  context  ratings  within  occupations 
to  the  patterns  across  occupations.  A simple  way  to  accomplish  this  is  to  identify  a small  subset 
of  occupations  for  which  we  have  relatively  large  samples  of  incumbents,  randomly  split  the 
samples  in  two  within  each  occupation,  and  then  correlate  the  mean  incumbent  ratings  within  and 
across  occupations.  To  ensure  that  the  correlations  are  not  inflated  due  to  large  differences  in 
scale  variances,  the  ratings  were  first  standardized  and  then  correlated.  This  comparison  was 
obtained  for  three  occupations:  First-Line  supervisors.  Secretaries  (Except  Legal  and  Medical), 
and  General  Office  Clerks. 

Table  8-17  presents  the  6x6  correlation  matrix  (3  occupations  x 2 samples  for  each).  This 
table  shows  that  the  only  within-occupation,  split-half  significant  correlation  (r  = .70)  is  for  First- 
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Line  Supervisors,  which  is  higher  than  any  of  the  cross-occupation  correlations.  Interestingly, 
most  of  the  correlations  between  First-Line  Supervisors  and  the  other  two  occupations  are 
negative.  This  may  suggest  that  First-Line  Supervisors  experience  the  organizational  context  very 
differently  than  Secretaries  and  Clerks. 

Analyses  of  Variance 

Another  method  of  examining  interrater  agreement  is  by  employing  an  analysis  of 
variance  (ANOVA),  where  the  descriptors  of  organizational  context  characteristics  are  treated  as 
repeated  measures  variables.  Tables  8-4a,  8-4b,  and  8-4c  present  the  ANOVA  results  for 
occupations,  organizations,  and  for  occupations  and  organizations,  respectively.  These  tables 
show  that  the  organizational  context  descriptors,  taken  together,  show  significant  differentiation 
between  occupations  and  between  organizations,  and  also  that  the  interaction  between 
occupations  and  organizations  is  important  above  and  beyond  each  alone  in  accounting  for  the 
variance  in  the  organizational  context  descriptors. 

The  analysis  of  variance  that  focused  on  occupations  (Table  8-4a)  shows  significant 
differences  among  occupations  and  a significant  effect  for  descriptor  by  occupation  interaction. 
The  analysis  of  variance  that  focused  on  organizations  (Table  8-4b)  shows  significant  differences 
among  organizations  and  a significant  effect  for  descriptor  by  organization  interaction.  The 
variance  attributed  to  organizations  is  somewhat  smaller  than  that  attributed  to  occupations. 
Finally,  the  analysis  of  variance  including  occupations,  organizations,  and  the  occupation  by 
organization  interaction  (Table  8-4c)  shows  significant  effects  for  the  descriptor  by  occupation 
interaction,  the  descriptor  by  organization  interaction,  and  the  descriptor  by  occupation  by 
organization  interaction. 
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The  interrater  reliability  coefficients  resulting  from  these  analyses  of  variance  are 
presented  in  Table  8-5.  As  was  true  of  the  data  reported  in  Tables  8-1  and  8-2,  agreement  across 
occupations  is  better  than  agreement  across  organizations. 

Tables  8-6a,  8-6b,  and  8-6c  present  the  results  of  a parallel  set  of  analyses  of  variance 
using  aggregate  descriptors  rather  than  the  individual  descriptors.  These  aggregates  were  based 
on  each  of  the  major  content  domains  in  our  content  model  (i.e.,  structure,  HR  practices,  goal 
setting,  roles,  leadership,  and  values).  The  aggregates  were  computed  by  standardizing  and 
averaging  the  variables  included  in  each  domain.  The  results  display  similar  patterns  to  the 
analyses  of  the  individual  descriptors,  but  the  aggregates  account  for  less  variance  than  the 
individual  descriptors.  Table  8-6c  shows  significant  interactions  for  aggregate  by  occupation, 
aggregate  by  organization,  and  aggregate  by  occupation  by  organization.  The  interrater  reliability 
coefficients  resulting  from  these  aggregate-level  analyses  of  variance  are  presented  in  Table  8-7. 
These  coefficients,  once  again,  support  the  finding  that  there  is  more  agreement  across 
occupations  than  across  organizations. 

Descriptor  and  Scale  Relationships 

Table  8-9a  presents  the  intercorrelations  of  all  of  the  organizational  context  descriptors 
and  scales  using  occupation-level  means  as  the  unit  of  analysis,  and  Table  8-9b  shows  these  same 
intercorrelations  based  on  organization-level  means.  Tables  8-10a  and  8-10b  present 
intercorrelations  of  descriptors  at  the  incumbent  level  (based  on  four  randomly  selected 
incumbents  from  each  occupation  or  each  organization,  respectively).  Because  the  emphasis  here 
is  on  occupational  description,  the  focus  of  this  discussion  will  be  on  Tables  8-9a  and  8-9b. 

Overall,  the  magnitude  and  direction  of  these  two  sets  of  correlations  appear  similar. 
However,  some  differences  are  worth  noting.  First,  the  organization  change  descriptors  (i.e.. 
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Number  of  Reorganizations  [Descriptor  #24]  and  Number  of  Changes  in  Job  Duties  [Descriptor 
#25])  were  negatively  correlated  with  Role  Negotiability  (Descriptor  #10),  Goal  Specificity 
(Descriptor  #1 1),  and  Goal  Feedback  (Descriptor  #12)  (-21,  -.16,  and  -.  14,  respectively)  at  the 
occupation  level,  while  positive  correlations  were  observed  at  the  organization  level  (.24,  .25, 
and  .19,  respectively).  Second,  the  job  rotation  policies  descriptor  was  negatively  correlated  to 
Empowerment  (Descriptor  #1)  and  Autonomy  (Descriptor  #2)  (-.22  and  -.25,  respectively)  at  the 
occupation  level,  but  showed  no  significant  correlations  at  the  organization  level.  Third,  the 
correlations  between  the  team  descriptors  (i.e.,  Number,  of  Teams  [Descriptor  #23]  and  % Time 
Spent  in  Teams  [Descriptor  #26])  and  Decentralization  [Descriptor  #13]  were  substantially 
higher  at  the  occupation  level  (.62  and  .34  compared  to  .14  and  .19)).  The  correlations  between 
the  team  descriptors  (Descriptors  #23  and  #26)  and  Empowerment  (Descriptor  #1)  and 
Autonomy  (Descriptor  #2)  were  also  substantially  higher  at  the  occupation  level  than  at  the 
organizational  level. 

Factor  Structure 

We  used  principal  components  analysis  with  a varimax  rotation  to  examine  the 
underlying  structure  of  the  incumbent  data.  Occupation  means  were  used  in  this  analysis,  in  part 
because  the  organizational  context  descriptors  appear  most  reliable  across  occupations  and  in 
part  because  occupation  means  were  used  for  factor  analyses  in  the  other  content  model  domains. 
Several  different  solutions  with  different  numbers  of  factors  were  examined,  and  the  four-factor 
solution  was  chosen  based  on  a scree  plot  of  the  eigenvalues  and  the  interpretability  of  the 
various  solutions;  this  rotated  factor  pattern  matrix  is  shown  in  Table  8-11.  These  four  factors 
were  labeled:  decentralization  and  employee  empowerment,  work  in  teams,  task-oriented 
leadership,  and  skill  variety. 
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Confirmatory  factor  analysis  was  conducted,  using  USREL  Vm,  to  confirm  the  structure 
of  the  organizational  context  content  model  (Joreskog  & Sorbom,  1993).  Occupation  means  were 
also  used  in  this  analysis.  Unfortunately,  the  model  did  not  converge  after  285  iterations.  The 
very  small  sample  size  (n  = 30)  used  in  this  analysis  can  perhaps  explain  this  result. 

Occupation  Differences 

Table  8-12  presents  the  means  and  standard  deviations  of  the  organizational  context 
measures  for  six  sample  occupations:  (1)  General  Managers  and  Top  executives,  (2)  Computer 
Programmers,  (3)  Registered  Nurses,  (4)  Police  Patrol  Officers,  (5)  Janitors  and  Cleaners,  and  (6) 
Maintenance  Repairers,  General  Utility.  As  expected,  some  organizational  characteristics  were 
found  to  be  associated  with  certain  occupations.  For  example.  General  Managers  and  Top 
Executives  showed  the  highest  levels  of  Empowerment  (Descriptor  #1),  Autonomy  (Descriptor 
Role  Conflict  (Descriptor  #7),  Decentralization  (Descriptor  #13),  and  the  largest  number  of 
training  topics  (Number  of  Training  Topics,  Descriptor  #28).  Computer  Programmers  indicated 
they  spent  the  most  time  working  in  teams.  Police  Patrol  Officers  experienced  the  least 
Empowerment  (Descriptor  #1),  Role  Negotiability  (Descriptor  #10),  and  Decentralization 
(Descriptor  #13). 

Discriminant  Analyses 

Two  sets  of  discriminant  analyses  were  conducted.  The  first  used  occupations  as  the  unit 
of  analysis,  as  was  done  for  the  discriminant  analyses  in  all  of  the  content  model  domains. 
However,  as  shown  in  Table  8-2,  the  organizational  context  descriptors  were  found  to,  in  general, 
yield  the  most  reliable  data  when  used  to  rate  the  occupation  within  organization  cells.  Thus,  we 
conducted  a second  set  of  discriminant  analyses  to  assess  how  well  these  descriptors  differentiate 


these  cells. 


Organizational  Context  8-17 


The  discriminant  analysis  for  occupations  provided  four  interpretable  functions,  and  these 
were  labeled:  (1)  decision  making  authority,  (2)  goal  setting,  (3)  team  structure,  and  (4)  skill 
variety.  Table  8- 13a  presents  the  correlations  between  the  organizational  context  descriptors  and 
each  of  these  functions  and  the  discriminating  variance  attributable  to  each  organizational 
context  descriptor.  These  four  functions  successfully  classified  28%  of  the  incumbents  into  the 
occupations  they  were  drawn  from.  Using  the  full  set  of  functions,  43%  of  incumbents  were 
classified  correctly. 

The  discriminant  analysis  across  the  occupation  within  organization  cells  yielded  five 
interpretable  functions,  and  these  were  labeled:  (1)  decision  making  authority.  (2)  organizational 
change.  (3)  goal  setting.  (4)  changes  in  supervision,  and  (5)  skill  variety.  These  five  functions 
successfully  classified  43%  of  the  incumbents  into  the  occupation  within  organization  cells  they 
were  drawn  from.  Using  all  30  functions,  65%  were  classified  correctly.  It  appears  that,  in  both 
analyses,  that  decision  making  authority  was  the  best  discriminating  function. 

Tables  8- 13a  and  8- 13b  also  show  the  sums  of  the  squared  discriminant  function 
coefficients  and  the  tj  coefficients  reflecting  the  variance  in  occupation  or  occupation  by 
organization  assignments  accounted  for  by  each  of  the  organizational  context  descriptors. 

In  addition  to  finding  that  the  classification  functions  derived  from  the  analysis  within 
organization  cells  yielded  higher  classification  rates.  Tables  8- 13a  and  8- 13b  provide  additional 
information  suggesting  that  the  organizational  context  descriptors  do,  in  fact,  discriminate  better 
across  the  occupation  within  organization  cells  than  they  do  across  occupations.  Specifically,  this 
information  consists  of  the  relevant  canonical  correlations  (Rc)  and  eigenvalues.  These 
correlations  represent  the  strength  of  association  between  the  relevant  occupational  groups  and 
the  set  of  organizational  context  variables.  The  mean  for  the  four  functions  from  the  across 
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occupation  analysis  (Table  8- 13a)  was  .56;  the  mean  Rc  for  the  five  functions  from  the  across 
occupation-within  organization  cells  analysis  was  .77. 

The  eigenvalues  represent  the  amount  of  between  groups  variance  explained  by  the  set  of 
organizational  context  descriptors.  Again,  eigenvalues  for  the  four  functions  from  the  across 
occupation  analysis  (M  = .49)  were  smaller  than  those  for  the  five  functions  from  the  within 
organization  cells  analysis  (M  = 1 .56).  Thus,  together  with  the  results  of  the  classification 
analyses,  these  findings  provide  evidence  that  the  organizational  context  descriptors  discriminate 
better  across  occupation-within  organization  cells  than  they  do  across  jobs. 

Organizational  Representative  Data  Analysis  Results 

Data  Cleaning 

Because  of  the  structure  of  the  CATI  interview,  several  strategies  were  required  for  data 
clean-up.  First,  the  CATI  included  three  options  for  invalid  responses:  don’t  know,  refuse  (to 
answer),  and  cannot  ascertain.  All  invalid  responses  were  set  to  missing.  In  addition,  the 
branching  structure  of  the  CATI  allowed  the  interviewers  to  skip  subsequent  branch-related 
questions,  resulting  in  missing  data  for  these  questions.  For  example,  respondents  who  reported 
that  they  did  not  have  selection  systems  were  not  asked  later  questions  related  to  selection 
systems  (e.g.,  how  many  selection  systems  are  based  on  occupation  analysis).  When  missing  data 
were  due  to  branching  in  the  survey,  it  was  replaced  with  the  appropriate  data.  For  example, 
respondents  who  reported  that  they  did  not  have  formal  training  programs  obviously  don’t  use 
any  ox  the  training  methods  listed  in  later  questions,  even  though  they  did  not  respond  to  these 
que ouons  (due  to  the  branching  in  the  CATI),  so  number  of  training  methods  was  set  to  zero  for 
these  respondents. 
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Several  financial  (i.e.,  establishment  and  organization  annual  revenue  and  annual  training 
budget)  and  organizational  change  (i.e.,  rightsizing  and  reorganization)  questions  had  high 
percentages  of  missing  data.  On  average,  the  financial  items  were  missing  for  37%  of  the 
establishments.  These  data  were  generally  missing  either  because  respondents  were  unable  or 
refused  to  provide  the  information.  The  measures  of  organizational  change  had,  on  the  average, 
26%  missing  data.  However,  many  of  the  change  items  asked  about  the  rate  of  organizational 
changes  over  the  previous  five  years,  and  were  only  asked  by  the  CATI  interviewers  if  the 
establishment  had  been  in  existence  for  at  least  five  years.  Consequently,  13%  of  the 
establishments  in  the  sample  did  not  answer  the  change  questions  simply  because  they  were  not 
asked.  The  financial  and  organizational  change  descriptors  are  not  included  in  most  of  the 
analyses  presented  here  in  order  to  maximize  the  available  sample  size. 

Descriptive  Statistics 

Most  of  the  variables  on  the  organizational  representative  survey  provided  continuous 
data  (e.g.,  rating  scales  ranging  from  1 to  7).  For  these  variables,  means  and  standard  deviations 
were  computed.  Table  8- 18a  presents  means  and  standard  deviations  for  those  descriptors  on 
which  the  metric  is  self  explanatory  (e.g.,  number  of  training  methods  used).  Table  8- 18b 
presents  means  and  standard  deviations,  along  with  the  rating  scale  used  to  collect  data 
concerning  each  descriptor,  for  those  descriptors  on  which  the  metric  requires  some  explanation. 
Table  8- 18c  presents  descriptive  statistics  for  the  organizational  values  descriptors.  Finally,  for 
some  of  the  descriptors,  the  answers  are  simply  yes  or  no,  and  frequencies  were  computed.  Table 
8-18d  presents  frequencies  for  these  variables. 
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Reliability 

For  the  a priori  scales  (e.g.,  scales  adapted  from  the  literature),  we  computed  internal 
consistency  reliability  coefficients.  For  the  organizational  representative  data,  only  one 
respondent  was  available  for  each  organization  (i.e.,  the  organizational  representative)  so 
interrater  reliability  could  not  be  assessed.  All  of  the  a priori  scales  included  in  the  CATI  had 
high  internal  consistency  reliabilities:  Formalization  (CATI  Item  #21  [5  parts],  a = .69), 
Standardization  (CATI  Item  #29  [14  parts],  a = .88),  Specialization  (CATI  Item  #28  [15  parts],  a 
= -67),  Decentralization  (CATI  Item  #30  [5  parts],  a = .79),  Information  Sharing  (CATI  Item  #31 
[5  parts],  a = .73),  use  of  contractors  (CATI  Items  #14  and  #15,  a = .79),  and  Use  of  Data  in 
Organizational  Decision-Making  (CATI  Item  #65  [6  parts],  a = .80).  The  items  that  measured 
changes  in  organizational  structure  were  divided  into  two  scales,  representing  two  types  of 
change:  rightsizing  (e.g.,  downsizing  and  removing  layers  of  managers)  and  reorganization  (e.g.. 
Number  of  Reorganizations  [CATI  Item  #25],  Number  of  Organizational  Chart  Revisions  [CATI 

Item  #27]).  Coefficient  alpha  was  .68  for  the  rightsizing  scale  and  .56  for  the  reorganization 
scale. 

Factor  Structure 

The  five  categories  that  represent  the  highest  level  of  the  content  model  for  organizational 
context  (structure,  organizational  values,  HR  practices,  goal  setting,  and  roles)  might  best  be 
viewed  as  a useful  heuristic  for  organizing  the  constructs  in  the  content  model,  but  they  were  not 
necessarily  expected  to  describe  the  empirical  structure  of  the  data.  Constructs  in  the  various 
categories  come  from  different  literatures  and,  in  some  cases,  even  different  disciplines,  so  there 
is  no  readily  available  theory  to  predict  or  explain  the  relationships  among  them.  However, 
within  each  of  these  broad  domains  there  is  a good  deal  of  information  in  the  literature 
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concerning  the  expected  structure.  Roles  were  only  measured  in  the  incumbent  questionnaire  and 
thus  not  included  in  this  analysis,  and  only  a few  questions  concerning  goals  were  included  in  the 
organizational  representative  questionnaire.  Therefore,  goal  items  were  included  in  the  factor 
analysis  of  HR  practices.  Descriptors  from  each  of  the  remaining  categories— structure  and 
values— were  factor  analyzed  separately. 

Only  half  of  the  organizational  representative  sample  (n  = 326)  was  used  to  conduct  these 
exploratory  factor  analyses.  The  remainder  of  the  sample  was  held  out  for  use  in  confirmatory 
analyses.  We  used  principal  factor  analysis  with  a varimax  rotation  to  examine  the  underlying 
structure  of  each  of  these  three  categories.  Only  cases  with  complete  data  were  included,  so 
sample  sizes  were  slightly  different  for  the  three  factor  analyses.  Several  different  solutions  with 
different  numbers  of  factors  were  examined,  and  one  solution  was  chosen  for  each  category 
based  on  a scree  plot  of  the  eigenvalues,  results  of  a parallel  analysis,  and  the  interpretability  of 
the  various  solutions. 

A six-factor  solution  was  selected  for  structure,  and  the  factor  pattern  is  shown  on  Table 
8- 19a.  These  factors  were  labeled:  (1)  use  of  teams.  (2)  formalization/standardization.  (3) 
information  sharing.  (4)  decentralization.  (5)  establishment  size,  and  (6)  organization  size.  Most 
of  the  structure  descriptors  load  on  at  least  one  of  these  factors  with  the  exception  of  Number  of 
Management  Levels  (Descriptor  #18,  a measure  of  hierarchy)  which  does  not  correlate  with  any 
of  the  other  structure  variables.  The  six  factors  account  for  40%  of  the  total  variance  in  these  data 
and  are  highly  interpretable;  most  of  the  variables  load  cleanly  on  only  one  factor. 

Table  8- 19b  shows  the  results  of  the  analysis  of  the  domain  of  human  resources  (HR) 
systems  and  practices.  The  four-factor  solution  was  selected.  These  factors  were  labeled:  (1) 
multiple  skill  training,  (2)  employee  benefits.  (3)  goal  setting,  and  (4)  high-performance  HR 
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practices.  Table  8-19c  shows  the  results  for  organizational  values.  The  four-factor  solution  was 
selected,  and  the  factors  were  labeled:  (1)  people  orientation.  (2)  risk  taking.  (3)  attention  to 
detail,  and  (4)  stability. 

In  order  to  examine  relationships  across  these  three  broad  domains,  and  to  determine 
whether  the  items  and  factors  thought  to  be  high-performance  do  in  fact  tend  to  co-exist,  we 
developed  a set  of  unit  weighted  composites  based  on  these  three  within-domain  factor  solutions. 
For  each  factor,  a composite  was  identified  that  included  all  variables  loading  .30  or  greater  on 
that  factor.  The  few  items  that  loaded  .30  or  greater  on  more  than  one  factor  were  assigned  to  the 
factor  on  which  the  loading  was  the  highest.  Variables  assigned  to  each  composite  were  then 
standardized  and  summed  to  form  composite  scores.  We  then  examined  the  intercorrelations 
among  scores  on  these  factor-based  composites  across  domains.  Table  8-20  presents  these  factor- 
based  composite  intercorrelations. 

Contrary  to  our  expectations  (based  on  the  literature),  size  was  not  strongly  related  to  any 
other  organizational  characteristic.  Measures  of  establishment  size  had  small  correlations  with 
the  formalization  composite  (r  = .20),  training  (r  = .26),  employee  benefits  (r  = .18),  and  goal 
setting  (r  = .17).  Organizational  size  correlated  .20  or  greater  with  only  one  other  organizational 
context  variable,  information  sharing  (r  = .20).  The  factor-based  composites  that  included 
variables  considered  high-performance  did,  in  fact,  show  moderate  intercorielations.  For 
example,  use  of  teams  was  correlated  with  information  sharing  (r  = .34),  decentralization  (r  = 
.33),  extent  of  training  (r  = .33),  and  goal  setting  (r  = .37).  Also,  information  sharing  was 
correlated  with  decentralization  (r  = .46),  extent  of  training  (r  = .36),  goal  setting  (r  = .43),  and 
risk-taking  values  (r  = .30). 
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In  order  to  farther  assess  the  interrelationships  among  these  three  content  domains, 
explore  the  overall  structure  of  these  data,  and  perhaps  identify  a high-performance  factor  across 
the  domains,  we  factor  analyzed  the  data  in  Table  8-20.  The  three  factor  solution  is  presented  in 
Table  8-21,  and  these  factors  were  labeled:  organizational  values,  high-performance  practices. 
and  establishment  size. 

As  mentioned  previously,  the  organizational  change  variables  were  not  included  in  the 
factor  analyses  because  of  missing  data,  so  relationships  between  these  variables  and  the  factor- 
based  composites  were  examined  separately.  The  correlation  between  the  two  organizational 
change  scales  (rightsizing  and  reorganization)  was  .57.  Interestingly,  even  though  these  two 
scales  are  fairly  highly  correlated,  when  correlated  with  the  factor-based  composites  measuring 
other  aspects  of  organizational  context,  the  two  scales  showed  different  correlation  patterns. 
Reorganization  was  positively  related  to  establishment  size  (r  = .22)  and  employee  benefits  (r  = 
.21).  Rightsizing,  however,  was  positively  related  to  decentralization  (r  = .22),  training  (r  = .21), 
and  employee  benefits  (r  = .24),  and  negatively  related  to  formalization/standardization  (r  = -.21). 

Confirmatory  Factor  Analysis 

Using  the  remainder  of  the  organizational  representative  sample,  which  was  held  out  from 
the  exploratory  factor  analyses  (N  = 335),  we  conducted  a second-order  confirmatory  factor 
analysis  to  test  the  hypothesis  that  a single  higher-order  construct,  which  might  be  labeled  “high- 
performance  business  practices,”  could  explain  the  relationships  between  decentralization, 
information  sharing,  use  of  teams,  use  of  data  in  decision  making,  risk-taking  values,  and 
multiple  skill  training.  We  used  LISREL  Vm  (Joreskog  & Sorbom,  1993)  to  test  this  model.  This 
model  was  selected  in  part  based  on  the  exploratory  factor  analysis  and  in  part  based  on  the 
literature  and  theory  concerning  high-performance  organizations.  The  indicators  for  each  of  these 


8-24  Arad,  Hanson,  & Schneider 


latent  variables,  as  well  as  the  factor  loadings  obtained  in  the  confirmatory  factor  analysis  are 
shown  in  Table  8-22.  Figure  8-4  presents  structural  coefficients  and  fit  indices  for  the  second- 
order  factor  analysis.  All  structural  coefficients  were  significant  and  a relatively  good  fit  was 
obtained  (GFI  = .92,  AGFI  = .90,  RMSR  = .06).  Use  of  teams  had  the  largest  loading  on  the 

higher-order  ‘high-performance”  factor  (.60),  while  risk-taking  values  had  the  smallest  loading 
(.37). 

We  also  developed  a “high-performance  business  practices”  factor  score  by  first 
multiplying  each  of  the  six  variables,  in  the  above  confirmatory  model,  by  its  loading.  We  then 
examined  correlations  between  this  composite  and  other  more  traditional  organizational 
characteristics.  The  high-performance  composite  did  not  correlate  significantly  with 
establishment  size.  However,  it  showed  small  positive  relationships  with  the  items  from  the 
CATI  that  measure  formalization  and  standardization  (r  = .16  and  r = .30,  respectively),  and 
somewhat  larger  relationships  with  extent  of  organizational  goal  setting  (r  = .46),  goal 
negotiability  (r  = .40),  and  compensation  and  benefits  (r  = .31  and  r = .32,  respectively). 
Interestingly,  high  performance  showed  small  but  positive  correlations  with  rightsizing  and 
reorganization  (r  = . 1 1 and  r = . 19,  respectively). 

Industry  Type  Comparisons 

Industry  type  was  included  in  the  0*NET  organizational  context  taxonomy  in  part 
because  the  literature  suggested  that  organizations  from  different  industries  would  differ 
systematically  in  terms  of  certain  organizational  characteristics.  Thus,  in  order  to  obtain  a 
preliminary  assessment  of  the  usefulness  of  the  variables  measured  in  the  CATI  for  describing 
organizations,  we  compared  scores  on  the  organizational  context  factor-based  composites  for 
organizations  from  different  types  of  industries.  We  used  multivariate  analysis  of  variance, 
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analysis  of  variance  and  t- tests  to  determine  whether  scores  on  the  14  factor-based  organizational 
context  composites  (defined  in  Tables  21a,  21b,  and  21c)  differed  significantly  for  organizations 
from  eight  major  industries:  construction,  manufacturing,  service,  finance,  retail  and  wholesale 
trade,  transportation  and  public  utilities,  public  administration,  and  high-technology  (SIC 
manual,  1987). 

The  MANOVA  for  industry  type  was  significant  and  7 of  the  14  factor-based 
organizational  characteristics  significantly  differentiated  among  different  industries  (see  Table  8- 
23).  Formalization/standardization  was  the  highest  for  organizations  in  public  administration  and 
the  lowest  in  wholesale  and  retail  organizations.  High  technology  and  finance  establishments 
indicated  the  highest  levels  of  information  sharing,  while  construction  companies  showed  the 
lowest  levels.  Establishment  size/specialization  was  the  highest  for  high  technology 
establishments  and  lowest  for  public  administration  and  construction  establishments.  High 
technology  and  public  administration  organizations  scored  the  highest  on  employee  benefits, 
while  those  involved  in  construction  scored  the  lowest.  High  technology  and  finance 
establishments  scored  the  highest  on  high-performance  HR  practices,  while  public  administration 
establishments  scored  the  lowest.  Finally,  people-oriented  values  were  the  most  important  in 
service  and  finance  establishments  and  the  least  important  in  manufacturing  and  public 
administration  establishments. 

Comparing  Incumbent  and  Organizational  Representative  Data 

Factor  Structure 

For  the  purpose  of  comparing  the  factor  structure  of  the  incumbent  data  with  that  of  the 
organizational  representative  data,  we  conducted  another  set  of  factor  analyses  using  the 
incumbent  data  designed  to  replicate,  to  the  extent  possible,  the  factor  analyses  that  were 
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conducted  using  the  organizational  representative  data.  Specifically,  we  used  organizations  as  the 
unit  of  analysis  (i.e.,  organizational  means  were  the  data  points),  whereas  in  the  factor  analyses  of 
incumbent  data  reported  previously  occupations  were  the  unit  of  analysis.  Also,  we  divided  the 
descriptors  according  to  the  three  broad  content  domains  that  were  used  in  the  organizational 
representative  factor  analyses.  However,  the  values  items  showed  poor  reliability  in  the 
incumbent  data,  so  only  two  domains  were  included  in  these  analyses:  HR  practices  (roles  and 
goals  were  included  here)  and  organizational  structure. 

We  used  principal  components  with  a varimax  rotation  for  these  analyses.  We  examined 
several  different  factor  solutions  for  each  domain,  and  selected  one  based  on  a scree  plot  of  the 
eigenvalues  and  the  interpretability  of  the  factor  solutions.  The  results  of  the  principal  component 
analysis  of  the  incumbent  data  by  content  domains  are  presented  in  Tables  8-24a  and  8-24b. 

Three  factors  were  chosen  for  organizational  structure  and  labeled:  decentralization  and 
employee  empowerment,  task-oriented  leadership,  and  work  in  teams.  Three  factors  were  also 
chosen  for  HR  practices  and  labeled:  HR  systems,  role  conflict,  and  individual  goal-setting. 

Comparing  the  factor  structure  of  the  incumbent  data  (Tables  8-24a  and  8-24b)  with  that 
of  the  organizational  representative  data  (Tables  8- 19a  and  8- 19b)  demonstrates  the  structural 
similarities  and  differences  between  the  two  data  sets.  In  the  organizational  structure  domain, 
both  data  sets  provide  support  for  decentralization  and  use  of  teams  as  two  distinct  factors.  In  the 
HR  practices  domain,  goal  setting  is  the  only  factor  common  to  both  data  sets.  However,  the 
incumbent  HR  systems  factor  (Table  8-24b)  incorporates  three  factors  from  the  organizational 
representative  factor  analysis:  multiple  skill  training,  employee  benefits,  and  high-performance 
HR  practices  (Table  8-1 9b). 
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This  comparison  also  highlights  the  complementary  relationships  between  the  structure  of 
the  two  data  sets.  Constructs  that  were  measured  only  at  the  incumbent  level  such  as  roles  and 
leadership,  were  identified  by  the  factor  analysis  of  the  incumbent  data.  Likewise,  constructs 
measured  only  at  the  organization  level  such  as  formalization,  standardization,  information 
sharing,  and  size  came  out  of  the  factor  analysis  of  the  organizational  representative  data. 

Cross-Level  Correlations 

In  order  to  examine  the  validity  of  some  of  our  descriptors  and  their  convergence  across 
levels  of  measurement,  we  correlated  several  organization-level  descriptors  from  the  incumbent 
questionnaire  with  their  parallel  descriptors/scales  from  the  organizational  representative 
questionnaire. 

We  expected  only  a subset  of  the  descriptors  to  correlate  across  levels.  Specifically,  we 
hypothesized  that  incumbent  measures  of  decentralization,  job  rotation  policies,  extent  of  work  in 
teams,  and  number  of  reorganizations  would  positively  correlate  with  measures  on  the 
organizational  representative  survey  targeting  these  same  constructs.  Also,  we  expected  measures 
of  individual  goal  setting  (i.e.,  goal  specificity  and  goal  feedback)  to  correlate  positively  with 
measures  of  organizational  goal  setting  (i.e.,  extent  of  goal  setting  and  goal  negotiability). 

However,  even  for  these  parallel  constructs  we  did  not  expect  to  find  strong  correlations 
between  the  two  levels  of  measurement  for  several  reasons.  First,  each  level  represents  a different 
perspective  or  source  of  information  about  the  relevant  construct.  Thus,  one  can  expect  that  each 
level  would  include  a substantial  unique  variance.  Moreover,  the  emphasis  in  the  organizational 
representative  questionnaire  was  on  the  establishment,  as  a whole,  while  the  incumbent 
questionnaire  focused  on  the  incumbent’s  specific  job.  As  a result,  organizational  representatives 
could  be  viewed  as  providing  information  about  the  establishment,  regardless  of  their  jobs  or 
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occupations,  whereas  the  incumbents  provided  information  about  the  specific  manifestation  of 
the  organizational  context  descriptors  in  their  particular  jobs.  The  second  reason  to  expect  only 
weak  relationships  concerns  the  sample  restrictions.  Due  to  the  sampling  strategy  used  (to  limit 
the  burden  on  any  single  establishment),  we  had  a limited  number  of  incumbent  responses  from 
any  given  organization,  and  these  incumbents  typically  represented  only  a few  occupations.  Thus, 
the  incumbent  data  cannot  be  viewed  as  representative  of  the  organization  as  a whole. 

The  correlations  between  the  incumbent  and  organizational  representative  measures  of 
decentralization  and  job  rotation  were  positive,  .27  and  .28,  respectively,  while  the  measures  of 
the  number  of  reorganizations  and  work  in  teams  did  not  correlate  significantly  across  levels. 
Surprisingly,  organizational  goal  setting  was  negatively  correlated  with  individual  goal 
specificity  and  goal  feedback  (r  = -.24  and  r = -.22  respectively),  and  goal  negotiability  (for  non- 
managers) was  negatively  correlated  to  individual  goal  feedback  (r  = -.24).  Perhaps  this  latter 
result  suggests  that  extensive  use  of  organizational  goal  setting  can  actually  hurt  the  quality  of 
goal  setting  practiced  by  individuals  (i.e.,  employees  and  managers)  in  the  organization. 

Conclusions 

The  purpose  of  the  analyses  described  here  was  to  evaluate  the  reliability  and  validity  of 
the  organizational  context  descriptors  included  in  the  0*NET  content  model.  The  descriptors  in 
this  domain  were  developed  for  the  purpose  of  describing  organizations,  so  one  primary  goal  in 
these  analyses  was  to  evaluate  their  usefulness  in  describing  organizations.  However,  because 
data  concerning  many  of  the  descriptors  were  collected  from  the  job  incumbents  in  the  0*NET 
prototype  sample,  we  were  also  able  to  assess  the  usefulness  of  this  subset  of  the  descriptors  for 
describing  occupations  and  also  for  describing  occupation  within  organization  “cells.”  Results  of 
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these  analyses  offer  preliminary  answers  to  several  key  questions  regarding  the  quality  of  the 
organizational  context  measures  and  the  methodology  used: 

1.  Can  the  0*NET  measures  be  used  to  collect  reliable  data  concerning  the  organizational 
context  within  which  occupations  occur? 

2.  Are  the  organizational  context  descriptors  useful  for  differentiating  among  organizations? 

3.  Are  the  organizational  context  descriptors  useful  for  differentiating  among  occupations? 

4.  Are  the  organizational  context  descriptors  useful  for  describing  and  gathering  information 
about  high-performance  organizations? 

5.  Do  the  data  support  the  structure  of  the  organizational  context  taxonomy? 

Reliability  of  the  Organizational  Context  Descriptors 
Data  concerning  several  of  the  organizational  context  descriptors  in  the  0*NET  content 
model  have  traditionally  been  collected  using  scales.  The  internal  consistency  reliability  of  the 
scales  included  in  both  the  incumbent  and  organizational  representative  surveys  was  generally 
quite  good,  and  comparable  to  that  obtained  using  similar  scales  in  previous  research  (see 
coefficient  alphas  in  Table  8-1). 

Interrater  reliability  could  only  be  assessed  for  the  data  collected  from  incumbents,  and 
these  data  were  assessed  in  terms  of  reliability  for  describing  occupations,  reliability  for 
describing  organizations,  and  reliability  for  describing  occupation  within  organization  cells.  As 
expected,  individual  descriptor  reliabilities  were  generally  highest  for  describing  occupation 
within  organization  cells,  suggesting  that  most  descriptors  capture  both  organizational  and 
occupational  differences.  In  addition,  the  majority  of  the  descriptors  showed  better  reliability  for 
describing  occupations  than  for  describing  organizations.  Only  a few  elements  of  organizational 
context,  as  rated  by  job  incumbents,  were  affected  more  by  the  incumbents’  organizational 
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affiliation  than  by  their  occupation  (e.g.,  adequate  resources,  organizational  change,  and  skill 
training). 

The  majority  of  the  descriptors  included  in  the  incumbent  questionnaire  showed  adequate 
interrater  reliability  for  describing  occupations  (sixteen  above  .80  for  30  raters,  nineteen  above 
.70).  The  fact  that  the  reliabilities  are  not  higher  may  be  because  organizational  context  is  not  as 
immediate  a part  of  incumbents’  day  to  day  working  environment,  or  perhaps  it  is  because  many 
of  the  descriptors  in  this  domain  are  quite  abstract.  A few  of  the  organizational  context 
characteristics  in  the  incumbent  questionnaire  had  generally  poor  interrater  reliability  (i.e.,  task 
identity,  feedback,  organizational  values,  leader  consideration,  and  visionary  leader). 
Accordingly,  we  excluded  these  descriptors  from  many  of  the  analyses  reported  here  and  are 
considering  dropping  some  of  them  from  future  0*NET  data  collection  efforts. 

In  general,  these  findings  suggest  that  the  manifestations  of  organizational  context 
characteristics  vary  across  occupations.  The  same  organizational  structure,  for  example,  may  be 
experienced  differently  by  a secretary,  an  engineer,  or  a top  executive.  It  seems  reasonable  to 
hypothesize  that  the  immediate  work  context  of  job  incumbents  (i.e.,  their  job  demands, 
requirements,  characteristics)  has  a large  impact  on  their  experience  of  most  organizational 
context  characteristics.  The  sample  available  for  the  analyses  described  here  did  not  allow  us  to 
systematically  assess  the  extent  to  which  respondents’  occupations  affect  their  perceptions  of 
organizational  characteristics,  siuce,  at  best,  data  from  incumbents  in  only  a few  occupations 
were  available  for  any  given  organization. 

The  theoretical  implications  of  these  findings  are  even  more  far-reaching  The  observed 
interactions  between  organizational  and  occupational  effects  highlight  the  shortcomings  of  most 
organizational  and  occupational  theories.  Traditionally,  organizational  theories  do  not  include 
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occupational  variables  nor  do  they  propose  how  organizational  variables  will  affect  individuals 
who  are  working  in  different  jobs.  Similarly,  occupational  theories  typically  ignore  the  effects  of 
organizational  context  on  job  requirements  and  characteristics.  The  findings  in  the  present 
research  emphasize  the  need  for  more  integration  between  these  two  disciplines. 

Differentiating  Among  Organizations  and  Occupations 
In  general,  the  organizational  context  descriptors  showed  sensible  differences  between 
organizations  as  well  as  between  occupations.  In  terms  of  organization  differences,  we  found 
expected  differences  between  organizations  representing  different  types  of  industries.  For 
example,  the  results  indicated  that  some  of  the  organizational  context  descriptors  show 
systematic  differences  between  high-technology  and  low-technology  (e.g.,  manufacturing, 
transportation,  construction,  retail)  establishments.  Specifically,  high-technology  establishments 
reported  more  information  sharing,  specialization,  types  of  employee  benefits,  and  high- 
performance  HR  practices  than  low-technology  establishments.  Public  ^ministration 
establishments,  on  the  other  hand,  had  higher  levels  of  formalization/standardization  and  more 
types  of  employee  benefits  than  other  types  of  establishments. 

The  organizational  context  data  collected  from  incumbents  had  mean  differences  across 
jobs  that  appear  systematic  and  sensible  as  well.  For  example,  workers  in  professional  and 
managerial  jobs  (e.g.,  general  managers  and  top  executives,  computer  programmers)  reported,  on 
average,  more  empowerment,  autonomy,  role  conflict,  decentralization,  and  a larger  number  of 
training  topics  than  did  those  in  more  traditionally  blue  collar  jobs  (e.g.,  janitors  and  cleaners, 
maintenance  repairers).  Also,  police  patrol  officers  reported  the  lowest  average  levels  of 
empowerment,  role  negotiability,  and  decentralization.  These  results  could  be  viewed  as 
reflecting  two  patterns  of  relationships  between  occupations  and  organizational  context: 
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occupation  level  exhibits  a positive  relationship  with  some  of  the  organizational  context 
descriptors  and,  as  suggested  before,  job  demands/activities/characteristics  often  overpower  or 
limit  the  impact  of  the  organizational  context. 

Describing  High-Performance  Organizations 

The  growing  interest  in  high-performance  organizations  was  at  the  core  of  the 
development  of  the  0*NET  content  model  for  the  organizational  context  domain.  One  of  the 
goals  in  this  area  was  to  describe  this  high-performance  phenomenon  and  collect  information 
concerning  the  business  practices  of  these  high-performance  organizations.  Indeed,  the 
exploratory  and  confirmatory  analyses  of  the  organizational  representative  data  provide  evidence 
for  the  existence  of  a high-performance  syndrome/phenomenon.  Many  of  the  characteristics  of 
high-performance  organizations,  identified  in  Arad,  Schneider,  and  Hanson  (1995),  appear  to  be 
captured  by  the  descriptors  in  this  domain  and  the  covariance  in  these  descriptors  can  be 
accounted  for  by  a single  higher-order  factor,  which  might  be  labeled  “high-performance 
practices.” 

Confirmatory  analyses  supported  the  hypothesis  that  the  high-performance  phenomenon 
includes  six  organizational  elements:  decentralization,  information  sharing,  use  of  teams,  use  of 
data  in  decision  making,  risk-taking  values,  and  multiple  skill  training,  which  are  consistent  with 
Westat’s  literature-based  list  of  high-performance  characteristics  (Westat,  1994).  Moreover,  use 
of  teams  appeared,  in  our  sample,  to  be  more  closely  tied  to  the  high-performance  phenomenon 
than  were  risk-taking  values. 

This  constellation  of  high-performance  practices  also  had  positive  relationships  with 
other  more  traditional  organizational  constructs.  Specifically,  high-performance  practices  were 
positively  associated  with  standardization  and  numbers  of  employee  benefits  and  compensation 
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elements.  Also,  high-performance  practices  were  positively  related  to  the  extent  to  which 
employees  are  required  to  set  goals  and  allowed  to  negotiate  their  goals. 

Structure  of  the  Organizational  Context  Data 

The  organizational  representative  data  provided  strong  support  for  the  0*NET 
organizational  context  taxonomy.  Within  the  HR  practices  and  structure  categories,  the  factors 
that  were  identified  resemble  the  constructs  included  in  the  0*NET  organizational  context 
taxonomy  (Arad,  Schneider,  & Hanson,  1995).  The  four  values  factors  identified  (i.e.,  people 
oriented,  risk  taking,  attention  to  detail,  and  stability)  make  a great  deal  of  sense,  and  partially 
replicate  previous  findings  (e.g.,  O’Reilly,  Chatman,  & Caldwell,  1991). 

Factor  analysis  results  for  the  incumbent  data  were  also  consistent  with  the  0*NET 
organizational  context  taxonomy.  When  occupational-level  data  were  used,  four  meaningful 
factors  were  identified:  employee  empowerment,  team  structure,  task-oriented  leadership,  and 
skill  variety.  Employee  empowerment,  team  structure,  and  skill  variety  are  three  structural 
variables  from  the  organizational  context  taxonomy,  while  task-oriented  leadership  is  one 
element  of  leadership  included  in  the  taxonomy.  Overall,  these  factors  suggest  that  some  aspects 
of  the  organizational  context  taxonomy  can  be  used  to  describe  and  differentiate  between 
occupations. 

The  discriminant  analyses  of  the  incumbent  data  did  not  completely  replicate  the  results 
of  the  factor  analyses,  but  it  did  show  a great  deal  of  similarity.  At  the  occupation-level,  four 
meaningful  functions  were  identified:  decision  making  authority,  goal  setting,  team  structure,  and 
skill  variety.  Decision  making  authority  and  goal  setting  taken  together  are  virtually  identical  to 
the  employee  empowerment  factor  from  the  factor  analysis;  team  structure  and  skill  variety  were 
identified  in  both  analyses.  The  discriminant  analysis  at  the  occupation  within  organization  cells 
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level  resulted  in  two  additional  unique  functions:  organizational  change  and  change  in 
supervision.  These  factors  provide  further  support  for  the  dual  nature  of  the  data  concerning 
organizational  context  descriptors  collected  from  incumbents,  and  suggest  that  descriptors 
measuring  change  were  more  affected  by  the  incumbents’  organisations  than  by  their  jobs. 

Comparisons  between  the  organizational  representative  and  incumbent  results  show  some 
structural  similarities.  In  both  data  sets,  use  of  teams,  decentralization/employee  empowerment, 
and  goal  setting  were  identified  as  distinct  elements.  Differences  between  the  factor  structure  of 
the  incumbent  and  organizational  representative  data  sets  should  not  be  overinterpreted,  since 
they  could  easily  be  explained  by  methodological  issues.  Specifically,  not  all  of  the 
organizational  context  descriptors  were  measured  at  both  levels.  The  incumbent  questionnaire 
included  some  constructs  that  were  not  measured  at  the  organizational  level,  such  as  roles  and 
job  characteristics;  while  the  organizational  representative  survey  included  some  constructs  that 
could  only  be  appropriately  measured  at  the  organization  level  (e.g.,  size,  formalization, 
specialization).  In  addition,  our  focus  was  on  collecting  as  much  data  as  possible  from  the 
organizational  representative,  so  the  incumbent  questionnaire  was  much  shorter  and  less 
comprehensive  that  the  organizational  representative  survey,  with  fewer  descriptors  per 
construct.  Even  so,  the  factor  analysis  results  for  these  two  data  sets  appear  complementary,  and 
taken  together  they  provide  strong  support  for  the  various  components  of  the  organizational 
context  taxonomy. 

As  expected,  the  correlations  between  parallel  descriptors  collected  from  organizational 
representatives  and  incumbents  were  generally  significant  but  not  large.  This  finding  suggests 
that  either  these  two  sources  of  organizational  information  do  in  fact  represent  two  distinct 
perspectives  or  that  our  samples  of  incumbents  were  not  sufficiently  representative  of  their 
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organizations  (or  both).  These  results,  along  with  the  factor  analyses  findings,  highlight  the 
importance  of  including  both  organizational  representative  and  incumbent  perspectives  in 
collecting  organizational  information,  although  our  understanding  of  the  similarities  and 
differences  between  these  two  perspectives  is  still  very  incomplete. 

Limitations 

The  limitations  of  the  incumbent  sample  used  in  these  analyses  should  be  kept  in  mind 
when  interpreting  the  results.  This  sample  included  a relatively  small  number  of  occupations 
within  each  organization,  by  design,  thus  restricting  our  ability  to  compare  sources  of  variance 
(e.g.,  occupations,  organizations)  more  rigorously.  Consequently,  only  general  trends  concerning 
the  effects  of  occupations  and  organizations  on  the  organizational  context  descriptors  could  be 
identified  in  the  incumbent  data. 

The  organizational  representative  sample  on  the  other  hand  was  quite  large,  representing 
over  600  organizations,  so  these  results  should  be  quite  robust.  The  fact  that  only  one  respondent 
was  surveyed  in  each  organization  limited  the  extent  to  which  we  could  evaluate  the  quality  of 
these  data,  but  the  fact  that  the  factor  structure  of  the  organizational  representative  data  is  highly 
interpretable  along  with  the  meaningful  differences  in  scores  obtained  across  industries  suggests 
that  these  data  are  generally  of  good  quality. 

Another  limitation  pertains  to  the  kind  of  analyses  used  in  this  study.  All  the  analyses 
were  designed  to  assess  the  internal  validity  of  the  descriptors  and  scales.  The  findings  provided 
preliminary  evidence  for  the  reliability  and  meaningfulness  of  the  descriptive  organizational 
context  information.  Cross-domain  and  other  external  analyses  are  needed  in  order  to  provide 
evidence  for  the  external  validity  of  the  measures. 
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Finally,  sampling  limitations  made  it  impossible  to  adequately  address  questions 
concerning  relationships  between  data  collected  from  the  organizational  representative  and  that 
collected  from  incumbents.  We  cannot  determine  the  extent  to  which  relationships  between 
parallel  organizational  context  constructs  measured  at  these  two  levels  are  attenuated  and/or 
confounded  by  the  limited  sample  of  occupations  available  from  each  sampled  organization. 
These  problems  also  make  it  difficult  to  assess  the  extent  to  which  other  occupational 
requirements  (e.g.,  skills)  are  affected  by  organizational  characteristics. 

Recommendations 

Results  of  the  present  analyses  show  that  the  organizational  context  data  are  adequately 
reliable,  have  an  empirical  structure  that  supports  the  0*NET  content  model,  and  appear  to  have 
potential  for  describing  both  occupations  and  organizations.  Thus,  we  recommend  that  the 
majority  of  these  descriptors  be  included  in  future  0*NET  data  collections.  However,  a few  of 
the  organizational  context  descriptors  were  problematic  and  should  be  dropped  from  the  surveys. 
In  the  organizational  representative  survey,  respondents  had  a great  deal  of  trouble  providing 
financial  data,  so  these  questions  should  be  excluded.  Several  of  the  variables  in  the  incumbent 
survey  had  low  interrater  reliabilities— organizational  values,  leader  consideration,  visionary 
leadership,  task  identity,  and  feedback-and  we  might  consider  dropping  these  variables  from  the 
future  data  collections.  Given  that  organizational  context  information  is  not  typically  included  in 
job  analysis  surveys,  it  is  not  too  surprising  that  some  of  these  descriptors  didn’t  perform  well, 
since  all  of  the  organizational  context  descriptors  could  be  viewed  as  somewhat  experimental  in 
this  context. 

Future  research  aimed  at  better  understanding  relationships  between  organizational  and 
occupational  variables  would  also  be  extremely  useful.  This  could  be  done  either  by  comparing 
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the  same  occupations  across  organizations,  or  different  occupations  within  the  same 
organization.  For  example  several  organizations  from  the  same  industry  that  are  made  up  of 
similar  occupations  could  be  identified,  including  some  that  are  known  to  use  high-performance 
business  practices  and  others  known  to  take  a more  traditional  approach.  A large  sample  of 
incumbents  from  each  of  these  organizations,  representing  a wide  variety  of  occupations,  could 
then  be  surveyed  concerning  organizational  context,  skills  and  other  relevant  descriptors,  in 
addition  to  collecting  the  organizational  information  from  one  or  more  organizational 
representatives.  These  data  would  allow  us  to  assess  the  relationships  between  organizational  and 
occupational  characteristics  with  a minimum  of  confounds.  It  is  unlikely  that  future  0*NET  data 
collections  will  provide  such  a data  set,  since  the  focus  is  on  collecting  data  concerning  a wide 
variety  of  occupations,  rather  than  in  depth  information  concerning  just  a few  organizations.  Still, 
this  sort  of  targeted  research  could  lead  to  a better  understanding  of  the  0*NET  descriptors  and 
aid  in  interpreting  data  collected  as  part  of  the  0*NET. 

The  present  research  provides  some  support  for  the  notion  that  many  organizations  do  in 
fact  employ  a constellation  of  business  practices  that  have  previously  been  labeled  “high- 
performance.”  Measures  of  many  of  the  relevant  variables  (e.g.,  use  of  teams,  risk-taking  values) 
have  been  developed  as  part  of  the  0*NET  project  and  preliminary  results  indicate  that  these 
measures  are  likely  to  be  useful  for  describing  organizations.  Further,  the  current  analyses 
provide  a great  deal  of  information  concerning  the  relationships  among  these  high-performance 
variables  as  well  as  their  relationships  with  other  important  organizational  variables.  However, 
this  research  did  not  address  the  critical  question  of  whether  these  practices  are  actually  related  to 
organizational  effectiveness.  The  survey  used  in  the  present  research  provides  an  excellent 
starting  point  for  future  research  aimed  at  understanding  the  relationships  between  high- 
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Figure  8-1 

Taxonomy  of  Organizational  Context  Variables 


Construct  Label 

Item  Number  on 
Incumbent 
Questionnaire 

Item  Number  in 
Organizational 
Representative  CATI 

Type  of  Industry 

Type  of  Industry  (i.e..  Organizational 
Output) 

(derived  from 
Standard  Industrial 
Classification 
Number) 

Structural  Characteristics 

Organizational  Structure 

Organization  and  Establishment  Size 

6, 7, 8, 9, 10, 12,  18, 
22 

Hierarchy 

Specialization 

28 

Formalization 

20, 21 

Standardization 

29 

Centralization  & Employee  Empowerment 

1, 2, 42a,  42b,  42c,  42d 

30 

Individual  versus  Team  Structure 

43 

32, 33, 34 

Type  of  Work  Teams 

35a,  35b,  35c,  35d, 
35e 

Skill  Variety 

5,  8,11 

Task  Significance 

6, 12, 17 

Task  Identity 

4,9, 14 

Autonomy 

3, 13, 16 

Feedback 

7,  10, 15 

Information  Sharing 

31 

Amount  of  Change  in  the  Organization 
Structure 

23, 24, 25, 26 

11, 13,23,24, 25,27 

Human  Resources  (HR)  Systems  & Practices 

Recruitment  Planning  and  Operations 

50 

56,  58 

Selection  Processes  and  Methods  Used 

49 

61 

ERIC 
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Figure  8-1  (continued) 

Taxonomy  of  Organizational  Context  Variables 


Construct  Label 

Item  Number  on 
Incumbent 
Questionnaire 

Item  Number  in 
Organizational 
Representative  CATI 

Group  and  Individualized  Socialization 

59 

Training  Topics/Programs  and  Methods 
Used 

46, 48 

45, 45a,  45b,  45c, 
45d,  45e,  45f,  45g, 
45h,  45i,  45j,  45k, 
48, 48a,  48b,  48c, 
48d,  48e,  48f,  48g, 
' 50a 

Extent/Support  of  Training 

51 

44,46,49a,  49b,  49c, 
53, 54 

Use  of  Data  in  Training  Process 

47 

Basis  of  Compensation 

66, 67, 68c,  68d,  68e, 
68f,  68g,  68h 

Benefit  and  Compensation  Elements 

44,45 

68a,  68b,  69a,  69b, 
69c,  69d,  69e,  69f, 
69g,  69h 

Use  of  Independent  Contractors 

14,  15 

Use  of  Data 

65 

Social  Processes  jj 

Culture  | 

Organizational  Values 

22a,  22b,  22c,  22d, 
22e,  22f,  22g,  22h,  22i, 
22j,  22k,  221 

70a-70bb 

Goals 

Goal  Specificity 

37, 38 

37, 38, 39, 41 

Goal  Feedback 

39, 40, 41 

43 

Goal  Negotiability 

40,42 

Roles 

Role  Conflict 

27, 28, 29,  30 

Role  Overload 

31,32, 33 

Role  Negotiability 

34, 35 
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Figure  8-1  (continued) 

Taxonomy  of  Organizational  Context  Variables 


Construct  Label 

Item  Number  on 
Incumbent 
Questionnaire 

Item  Number  in 
Organizational 
Representative  CATI 

Leadership 

Consideration 

18 

T ask-Orientation 

19 

Visionary 

20 

Problem  Solving 

21 

Changes  in  Leadership 

23 

Note.  See  Appendix  F of  Development  of  Prototype  Occupational  Information  Network 
(Q*NET)  Content  Model  (Peterson,  Mumford,  Borman,  Jeanneret,  & Fleishman,  1995)  for  the 
exact  wording  of  the  incumbent  questionnaire  items.  See  Appendix  A of  this  report  for  the  exact 
wording  of  the  organizational  representative  CATI  items.  The  constructs  in  the  above  taxonomy 
differ  slightly  from  those  in  the  original  taxonomy  described  by  Arad,  Schneider,  and  Hanson 
(1995);  the  taxonomy  has  been  modified,  as  described  in  the  text,  based  in  part  on  feedback  from 
organizational  representatives. 
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Figure  8-2 

Description  and  Definitions  of  Organizational  Context  Descriptors  and  Scales  (Incumbent  Questionnaire) 


Descriptor/Scale  (#  Items) 

Definition 

Item  # (s)  in  Questionnaire 

1.  Empowerment  (2) 

Has  influence  and  control  over  decisions  made 
in  unit  or  department 

1,2 

2.  Autonomy  (3) 

Has  freedom  and  autonomy  in  job 

3,  13,  16 

3.  Task  Identity  (3) 

The  job  involves  doing  a ‘whole*  and 
identifiable  piece  of  work 

4,  9,  14 

4.  Skill  Variety  (3) 

The  job  requires  one  to  use  many  different 
skills  and  talents 

5,8,11 

5.  Task  Significance  (3) 

The  job  is  important  and  affects  the  lives  or 
well-being  of  other  people 

6,  12,  17 

6.  Feedback  (3) 

The  job  itself  provides  information  about 
one’s  performance 

7,  10, 15 

7.  Role  Conflict  (4) 

Works  with  supervisors  or  groups  that  have 
conflicting  requests 

27,  28,  29,  30 

8.  Role  Overload  (2) 

Does  not  have  enough  time  to  accomplish 
assigned  work 

32,  33 

9.  Adequate  Resources 

Has  adequate  resources  and  materials  to 
complete  assignments 

31 

10.  Role  Negotiability  (2) 

Can  negotiate  the  nature  of  role/job  with 
supervisor 

34,  35 

1 1 . Goal  Specificity  (2) 

Has  individual  goals  that  are  specific  and 
quantitative 

37,  38 

12.  Goal  Feedback  (3) 

Receives  feedback  on  goal  achievement 

39, 40, 41 

13.  Decentralization  (4) 

Has  the  authority  to  monitor  quality  data, 
determine  work  flow,  invest  in  new  equipment, 
and  develop  new  products 

42a,  42b,  42c,  42d 

14.  People-Oriented  Values  (4) 

Importance  to  the  organization  of  the  values 
that  emphasize  caring  about  employees, 
fairness,  and  openness 

22b,  22f,  22i,  22k 

15.  Risk-Taking  Values  (6) 

Importance  to  the  organization  of  the  value" 
that  emphasize  taking  chances,  innovation, 
flexibility  and  aggressiveness 

22a,  22d,  22e,  22g,  22h,  221 

Figure  8-2  (continued) 

Description  and  Definitions  of  Organizational  Context  Descriptors  and  Scales  (Incumbent  Questionnaire’) 


Descriptor/Scale  (#  Items) 

Definition 

Item  # (s)  in  Questionnaire 

16.  Attention  to  Details  Values  (2) 

Importance  to  the  organization  of  the  values 
that  emphasize  precision  and  quality  are 
important 

22c,  22j 

17.  Stability  Value 

Importance  to  the  organization  of  stability 

22d 

18.  Leader:  Consideration 

Immediate  supervisor  acts  in  friendly  and 
supportive  manner 

18 

19.  Leader:  Task  Orientation 

Immediate  supervisor  sets  goals  and  assigns 
tasks  for  the  work  group 

19 

20.  Leader:  Visionary 

Immediate  supervisor  provides  a clear  vision 
for  the  work  group  and  keeps  everybody 
committed 

20 

21.  Leader:  Problem  Solving 

Immediate  supervisor  solves  difficult 
problems  quickly  and  effectively 

21 

22.  No.  Supervisors 

Number  of  supervisors  in  the  past  year 

23 

23.  No.  Teams 

Number  of  work  teams  belonged  to  in  the  past 
year 

24 

24.  No.  Reorganizations 

Number  of  reorganizations  of  primary  work 
group 

25 

25.  No.  Changes  in  Job  Duties 

Number  of  times  the  nature  of  job  duties 
changed  dramatically 

26 

26.  % Time  Spent  in  Teams 

Percent  of  time  spent  working  in  an  intact 
team 

43 

27.  No.  Training  Methods 

Number  of  training  methods  used  in  company 
training  courses  attended  in  the  last  two  years 

48 

28.  No.  Training  Topics 

Number  of  content  areas  of  formal  training 
received  in  the  last  two  years 

46 

29.  No.  Selection  Methods 

Number  of  assessment  methods  used  to  select 
people  for  this  job 

49 

30.  No.  Recruitment  Sources 

Number  of  sources  used  to  recruit  people  for 
this  job 

50 

31.  No.  Compensation  Elements 

Number  of  elements  included  in  the 
compensation  package  for  this  job 

44 
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Figure  8-2  (continued) 

Description  and  Definitions  of  Organizational  Context  Descriptors  and  Scales  (Incumbent  Questionnaire^ 


Descriptor/Scale  (#  Items) 

Definition 

Item  # (s)  in  Questionnaire 

32.  No.  Benefit  Elements 

Number  of  elements  included  in  the  benefits 
package  for  this  job 

45 

33.  Job  Rotation  Practices 

The  scope  of  job  rotation  practices  in  this  job 

51 

Note.  See  Appendix  F of  Development  of  Prototype  Occupational  Information  Network  (Q*NED  Content  Model 
(Peterson,  Mumford,  Borman,  Jeanneret,  & Fleishman,  1995)  for  the  exact  wording  of  the  incumbent  questionnaire 
items. 
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Figure  8-3 

Thirty  Occupations  With  Four  or  More  Incumbents  Completing  the  Organizational  Context 
Questionnaire 


Occupation  Code 

Occupation  Tide 

Number  of 
Respondents 

15005 

Education  Administrators 

8 

19005 

General  Managers  & Top  Executives 

40 

22114 

Chemical  Engineers 

4 

25105 

Computer  Programmers 

9 

27311 

Recreation  Workers 

6 

31302 

Teachers,  Preschool 

5 

31305 

Teachers,  Elementary  School 

5 

32502 

Registered  Nurses 

27 

32902 

Medical  $ Clinical  Laboratory  Technologists 

4 

49008 

Salespersons,  Except  Scientific  & Retail 

11 

49011 

Salespersons,  Retail 

15  1 

49021 

Stock  Clerks,  Sales  Floor 

7 1 

49023 

Cashiers 

15 

51002 

First  line  Supervisors,  Clerical/Administrative 

50 

55108 

Secretaries,  Except  Legal  & Medical 

64 

55305 

Receptionists  & Information  clerks 

12 

55338 

Bookkeeping,  Accounting,  & Auditing  Clerks 

20 

55347 

General  Office  Clerks 

68 

61005 

Police  & Detective  Supervisors 

12 

63014 

Police  Patrol  Officers 

20 

65008 

Waiters  & Waitresses 

12 

65038 

Food  Preparation  Workers 

29 

66008 

Nursing  Aids,  Orderlies,  & Attendants 

17 

67005 

Janitors  & Cleaners 

21 

85119 

Other  Machinery  Maintenance  Mechanics 

5 

85132 

Maintenance  Repairer,  General  Utility 

30 

87902 

Earth  Drillers,  Except  Oil  & Gas 

6 

92974 

Packing  & Filling  Machine  Operators 

12 

97102 

Truck  Drivers,  Heavy  or  Tractor  Trailers 

9 

97111 

Bus  Drivers,  School 

11 

820 


Figure  8-4 

Model  and  Structural  Coefficients  for  Second-Order  Confirmatory  Factor  Analysis:  Organizational  Context 
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Table  8-2 

Reliability  of  Rated  Differences  between  Occupations  and  Organizations  Considering 
Varying  Numbers  of  Raters:  Organizational  Context 


Number  of  Raters  for  Each  Level  of  Analysis 


Across 


Across 


Within 


Descriptor/Scale 

Dd 

, e 

Do 

Ii 

Do 

r i 

Do 

1.  Empowerment 

25 

91 

05 

63 

23 

90 

2.  Autonomy 

12 

81 

02 

42 

21 

89 

3.  Task  Identity 

03 

51 

00 

00 

04 

57 

4.  Skill  Variety 

24 

91 

20 

88 

21 

89 

5.  Task  Significance 

17 

86 

14 

82 

13 

82 

6.  Feedback 

03 

46 

00 

00 

01 

30 

7.  Role  Conflict 

07 

68 

07 

69 

06 

64 

8.  Role  Overload 

06 

66 

11 

80 

14 

82 

9.  Adequate  Resources 

08 

71 

05 

59 

11 

79 

10.  Role  Negotiability 

11 

78 

04 

58 

07 

68 

1 1.  Goal  Specificity 

27 

92 

13 

82 

30 

93 

12.  Goal  Feedback 

15 

84 

08 

72 

21 

89 

13.  Decentralization 

32 

93 

08 

73 

23 

90 

14.  People-Oriented  Values 

00 

00 

03 

44 

06 

67 

15.  Risk-Taking  Values 

00 

00 

05 

60 

08 

72 

16.  Attention  to  Detail  Values 

02 

41 

02 

38 

11 

78 

17.  Stability  Value 

00 

00 

05 

62 

05 

63 

18.  Leader:  Consideration 

04 

54 

02 

33 

03 

52 

19.  Leader:  Task  Orientation 

06 

67 

06 

66 

21 

89 

20.  Leader:  Visionary 

02 

43 

00 

00 

03 

49 

2 1 . Leader:  Problem  Solving 

08 

73 

01 

18 

18 

87 

22.  No.  Supervisors 

20 

88 

11 

79 

30 

93 

23.  No.  Teams 

28 

92 

04 

58 

15 

84 

24.  No.  Reorganizations 

04 

52 

06 

64 

21 

89 

25.  No.  Changes  in  Job  Duties 

02 

37 

09 

74 

16 

85 

26.  % Time  Spent  in  Teams 

23 

90 

10 

77 

16 

85 

27.  No.  Training  Methods 

17 

86 

17 

86 

37 

95 

28.  No.  Training  Topics 

20 

88 

16 

85 

39 

95 

29.  No.  Selection  Methods 

17 

86 

09 

75 

27 

92 

30.  No.  Recruitment  Sources 

05 

61 

11 

78 

12 

80 

31.  No.  Compensation  Elements  19 

87 

17 

86 

28 

92 

32.  No.  Benefit  Elements 

22 

89 

15 

84 

39 

95 

33.  Job  Rotation  Practices 

13 

82 

06 

65 

13 

82 

a Reliability  estimates  across  occupations  are  based  on  30  occupations  with  Organizational 
Context  questionnaire  responses  from  at  least  4 incumbents  (mean  number  of  incumbents  = 
18.47,  median  = 12,  harmonic  mean  = 10.14). 
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Table  8-2  (continued) 

Reliability  of  Rated  Differences  between  Occupations  and  Organizations  Considering 
Varying  Numbers  of  Raters:  Organizational  Context 

bReliability  estimates  across  organizations  are  based  on  70  organizations  with  Organizational 
Context  questionnaire  responses  from  at  least  4 incumbents  (mean  number  of  incumbents  = 7.91, 
median  = 7.00,  harmonic  mean  = 6.51).  Decimals  are  omitted. 

Statistics  are  based  on  50  cells  where  a cell  is  defined  as  incumbents  in  the  same  occupation  and 
the  same  organization  with  Organizational  Context  questionnaire  responses  from  at  least  4 
incumbents  per  cell  (mean  number  of  incumbents  = 4.85,  median  = 4.00,  harmonic  mean  = 4.72). 
dS ingle  rater  estimates  of  reliability  were  obtained  by  calculating  the  intraclass  correlation  for 
single  judge  ratings  across  occupations:  ICC(1,1)  = rBMS-WMSl/fBMS+fk- 1 VWMS1  (Shrout  & 
Fleiss,  1979),  where  k is  the  harmonic  mean  of  the  number  of  ratings  provided  on  each 
occupation. 

^Estimates  of  reliability  for  30  raters  were  obtained  by  applying  the  Spearman-Brown  correction 
formula  to  the  single  rater  reliability  estimates. 


O 

ERIC 


Table  8-4a 


Vi  * uj.a«aiw  1U1  v_/ccupcuiun,  dilU 

Sources  of  Variation:  Organizational  Context 

rvcicvani  uuera 
MS 

cuons  as 
F 

Source  of  Variation 

SS 

df 

Occupations 

580.22 

29 

20.01 

3.10* 

S (Occupations) 

3379.80 

524 

6.45 

Descriptor 

6830.05 

31 

220.32 

173.42* 

Descriptor  x Occupations 

3163.87 

899 

3.52 

2.77* 

Descriptor  x S (Occupations)  20637.13 

16244 

1.27 

Note.  Occupations  are  treated  as  random,  between-subjects  effects,  while 
descriptors  are  treated  as  fixed,  within-subjects  effects. 

*E  < 05 


O 

ERiC 
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Table  8-4b 


j — : — — : : : — — — v 

of  Variation:  Organizational  Context 

^ ******  AWAV  V 

Source  of  Variation 

SS 

df 

MS 

F 

Organization 

691.61 

69 

10.02 

1.48* 

S(Organizations) 

3268.41 

484 

6.75 

Descriptor 

11041.27 

31 

356.17 

282.35* 

Descriptor  x Organizations 

4874.01 

2139 

2.28 

1.81* 

Descriptor  x S (Organizations) 

18926.99 

15004 

1.26 

Note.  Organizations  are  treated  as  random,  between-subjects  effects,  while  descriptors 
are  treated  as  fixed,  within-subjects  effects. 

*E  < .05 


Table  8-4c 

Analysis  of  Variance  for  Descriptor.  Occupation.  Organization,  and  Relevant 
Interactions  as  Sources  of  Variation:  Organizational  Context 


Source  of  Variation 

SS 

df 

MS 

F 

Occupations 

434.82 

27 

16.10 

2.82* 

Organizations 

511.71 

67 

7.64 

N.S 

Occupations  x Organizations 

948.24 

129 

7.35 

1.29* 

S (Occupations)  x S(Organizations) 

1874.18 

328 

5.71 

Descriptor 

5860.59 

31 

189.05 

166.77* 

Descriptor  x Occupations 

2078.07 

837 

2.48 

2.19* 

Descriptor  x Organizations 

3522.54 

2077 

1.79 

1.50* 

Descriptor  x Occupations  x Organizations 

5250.45 

3999 

1.31 

1.16* 

Descriptor  x S(Occupations)  x S(Organizations)  11526.79 

10168 

1.13 

Note.  Occupations  and  organizations  are  treated  as  random,  between-subjects  effects,  while 
descriptors  are  treated  as  fixed,  within-subjects  effects. 

*E  < -05 


Table  8-5 

Interrater  Agreement  Coefficients  Across  Occupations  and  Across  Organizations:  Organizational 
Context 


Number  of  Raters  for  Each  Level  of  Analysis 
Scale  Type  Ik  D Do 

Across  Occupations3  64  15  84 

Across  Organizations15  45  11  79 


Note.  Full  sample  interrater  agreement  coefficients  (&)  were  obtained  by  considering  the 
“Descriptor  x Occupations”  terms  from  Tables  8-4a  and  8-4b  as  true  variance.  Error  variance  was 
defined  as  the  “Descriptor  x S(Occupations)”  term.  Estimates  of  reliability  for  1 and  30  raters 
were  obtained  by  applying  the  Spearman-Brown  correction  formula  to  the  k rater  reliability 
estimates,  where  k is  the  harmonic  mean  of  the  number  of  raters  for  each  occupation.  Decimals 
are  omitted. 

aInterrater  agreement  coefficient  estimates  are  based  on  30  occupations  with  Organizational 
Context  questionnaire  responses  from  at  least  4 incumbents  (mean  number  of  incumbents=  18.47, 
median=12,  harmonic  mean=10.14). 

bInterrater  agreement  coefficient  estimates  are  based  on  70  organizations  with  Organizational 
Context  questionnaire  responses  from  at  least  4 incumbents  (mean  number  of  incumbents=  7.91, 
median=7.00,  harmonic  mean=6.51). 


3 
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Table  8-6a 


nn  Cl ■ ' 

Interactions  as  Sources  of  Variation:  Organizational  Context 

F 

Source  of  Variation 

SS 

df 

MS 

Occupations 

75.49 

29 

2.60 

2.37* 

S(Occupations) 

575.93 

524 

1.10 

Aggregate 

1.06 

5 

0.21 

N.S. 

Aggregate  x Occupations 

129.60 

145 

0.89 

2.51* 

Aggregate  x S (Occupations) 

931.11 

2620 

0.36 

Note.  Occupations  are  treated  as  random,  between-subjects  effects,  while 
aggregate  descriptors  are  treated  as  fixed,  within-subjects  effects. 

*E  < .05 


Table  8-6b 

Analysis  of  Variance  for  Ageregate  Descriptor.  Organization,  and  Relevant 
Interactions  as  Sources  of  Variation:  Organizational  Context 


Source  of  Variation 

SS 

df 

MS 

F 

Organization 

101.08 

69 

1.46 

N.S. 

S (Organizations) 

550.33 

484 

1.14 

Aggregate 

0.15 

5 

0.03 

N.S. 

Aggregate  x Organizations 

212.08 

345 

0.61 

1.75* 

Aggregate  x S (Organizations) 

848.63 

2420 

0.35 

Note.  Organizations  are  treated  as  random,  between-subjects  effects,  while 
aggregate  descriptors  are  treated  as  fixed,  within-subjects  effects. 

*P<  .05 


Table  8-6c 

Analysis  of  Variance  for  Aggregate  Descriptor.  Occupation,  Organization,  and  Relevant 
Interactions  as  Sources  of  Variation:  Organizational  Context 


Source  of  Variation 

SS 

df 

MS 

F 

Occupations 

61.20 

27 

2.27 

2.24* 

Organizations 

88.95 

67 

1.33 

NS. 

Occupations  x Organizations 

158.09 

129 

1.23 

NS. 

S(Occupations)  x S(Organizations) 

332.32 

328 

1.01 

Aggregate 

0.57 

5 

0.11 

NS. 

Aggregate  x Occupations 

106.94 

135 

0.79 

2.56* 

Aggregate  x Organizations 

175.94 

335 

0.53 

1.70* 

Aggregate  x Occupations  x 
Organizations 

239.98 

645 

0.37 

1.20* 

Aggregate  x S(Occupations)  x S 
(Organizations) 

506.59 

1640 

0.31 

Note.  Occupations  and  organizations  are  treated  as  random,  between-subjects  effects, 
while  aggregate  descriptors  are  treated  as  fixed,  within-subjects  effects. 

*p  < .05 


Table  8-7 

Interrater  Agreement  Coefficients  for  Aggregate  Descriptors  Across 
Occupations  and  Across  Organizations:  Organizational  Context 

Number  of  Raters  for 
Each  Level  of  Analysis 
Ik  Ii  Do 

Across  Occupations* 2 * *  60  13  82 

Across  Organizations'5  41  10  77 


Note.  Full  sample  interrater  agreement  coefficients  (i*)  were  obtained  by 
considering  the  “Aggregate  x Occupations  or  Organizations”  terms  from 
Tables  8-6a  and  8-6b  as  true  variance.  Error  variance  was  defined  as  the 
“Aggregate  x S(Occupations)  or  S(Organizations)”  term.  Estimates  of 
reliability  for  1 and  30  raters  were  obtained  by  applying  the  Spearman-Brown 
correction  formula  to  the  k rater  reliability  estimates,  where  k is  the  harmonic 
mean  of  the  number  of  raters  for  each  occupation.  Decimals  are  omitted. 

2 Interrater  agreement  coefficient  estimates  are  based  on  30  occupations  with 

Organizational  Context  questionnaire  responses  from  at  least  4 incumbents 

(mean  number  of  incumbents=  18.47,  median=12,  harmonic  mean=10.14). 
b Interrater  agreement  coefficient  estimates  are  based  on  70  organizations  with 

Organizational  Context  questionnaire  responses  from  at  least  4 incumbents 
(mean  number  of  incumbents=7.91,  median=7.00,  harmonic  mean=6.51). 


Table  8-9a 

Intercorrelations  of  Descriptors/Scales  (Occupation-Level  Data):  Organizational  Context 
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Table  8-9a  (continued) 

Intercorrelations  of  Descriptors/Scales  (Occupation-Level  Data):  Organizational  Context 
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Table  8-9b 

Intercorrelations  of  Descriptors/Scales  (Organization-Level  Data):  Organizational  Context 
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Table  8-9b  (continued) 

Intercorrelations  of  Descriptors/Scales  (Organization-Level  Data):  Organizational  Context 
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Table  8-9b  (continued) 

Intercorrelations  of  Descriptors/Scales  (Organization-Level  Data):  Organizational  Context 
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Table  8-  10  a 

Intercorrelations  of  Descriptors/Scales  (Individual-Level  Data):  Organizational  Context 
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Table  8-10a  (continued) 

Intercorrelations  of  Descriptors/Scales  (Individual-Level  Data1):  Organizational  Context 
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Table  8- 10a  (continued) 

Intercorrelations  of  Descriptors/Scales  (Individual-Level  Data):  Organizational  Context 
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Table  8- 10b 

Intercorrelations  of  Descriptors/Scales  (Individual-Level  Data):  Organizational  Context 
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Table  8- 10b  (continued) 

Intercorrelations  of  Descriptors/Scales  (Individual-Level  Data):  Organizational  Context 
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Table  8-11 


Principal  Components  Analysis  Pattern  Matrix:  Organizational  Context 

Factor 


Descriptor/Scale 

FI 

F2 

F3 

F4 

Communality 

1.  Empowerment 

.80 

.07 

-.12 

.18 

.69 

2.  Autonomy 

.72 

-.15 

-.21 

.30 

.68 

3.  Skill  Variety 

.38 

.12 

-.16 

.78 

.80 

4.  Task  Significance 

-.02 

-.02 

.05 

.74 

.55 

5.  Role  Conflict 

.26 

.64 

.00 

.27 

.54 

6.  Role  Overload 

.41 

.17 

-.57 

-.07 

.53 

7.  Adequate  Resources 

-.47 

-.33 

-.02 

-.14 

.35 

8.  Role  Negotiability 

.44 

-.17 

-.11 

-.16 

.26 

9.  Goal  Specificity 

.74 

-.03 

-.17 

-.06 

.58 

10.  Goal  Feedback 

.80 

-.16 

-.01 

-.05 

.66 

1 1 . Decentralization 

.67 

.44 

-.32 

.08 

.75 

12.  Leader:  Task  Orientation 

.01 

-.26 

.68 

-.08 

.53 

13.  Leader:  Problem  Solving 

-.12 

.06 

.65 

-.26 

.51 

14.  No.  Supervisors 

-.24 

.44 

.71 

-.20 

.79 

15.  No.  Teams 

.46 

.63 

.00 

.16 

.64 

16.  No.  Reorganizations 

-.14 

.76 

.07 

-.08 

.62 

17.  No.  Changes  in  Job  Duties 

-.21 

.70 

.22 

-.09 

.58 

18.  % Time  Spent  in  Teams 

-.02 

.74 

-.09 

.01 

.56 

19.  No.  Training  Topics 

.48 

.15 

-.22 

.29 

.39 

20.  No.  Training  Methods 

.05 

.14 

-.50 

.45 

.48 

2 1 . No.  Selection  Methods 

-.02 

.01 

-.26 

.87 

.82 

23.  No.  Compensation  Elements  .43 

.03 

-.57 

-.42 

.68 

24.  No.  Benefits  Elements 

.24 

-.34 

-.64 

.04 

.59 

25.  Job  Rotation  Practices 

-.37 

.57 

-.44 

-.16 

.69 

Percent  of  Variance 

18 

15 

13 

11 

Eigenvalue 

4.60 

3.80 

3.36 

2.85 

Note.  N = 30.  The  correlation  matrix  was  based  on  means  calculated  at  the  occupation 
level.  FI  = Decentralization/Employee  Empowerment;  F2  = Work  in  Teams;  F3  = 
Task-Oriented  Leadership;  F4  = Skill  Variety.  These  loadings  are  based  on  orthogonal 
varimax  rotation. 


Table  8-12 

Descriptor  Means  and  Standard  Deviations  on  Six  Example  Jobs:  Organizational  Context 
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Table  8-12  (continued) 

descriptor  Means  and  Standard  Deviations  on  Six  Example  Jobs:  Organizational  Context 
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Table  8- 13a 

Rotated  Correlations  Between  Discriminating  Variables  and  Canonical  Discriminant  Functions 
Across  Occupations:  Organizational  Context 


Functions 


Descriptor/Scale 

FI 

F2 

F3 

F4 

IF2 

„2 

T\ 

1 . Empowerment 

.26 

.42 

.16 

-.09 

.28 

.20 

2.  Autonomy 

.10 

.26 

-.04 

.26 

.15 

.12 

4.  Skill  Variety 

.22 

.00 

.22 

.51 

.36 

.19 

5.  Task  Significance 

-.03 

-.05 

.26 

.34 

.19 

.15 

7.  Role  Conflict 

.02 

.15 

.26 

.05 

.09 

.09 

8.  Role  Overload 

.26 

.03 

.15 

-.08 

.10 

.09 

9.  Adequate  Resources 

.03 

-.25 

-.06 

.01 

.07 

.09 

10.  Role  Negotiability 

.19 

.15 

-.09 

-.09 

.08 

.11 

1 1 . Goal  Specificity 

.09 

.53 

.11 

.11 

.31 

.21 

12.  Goal  Feedback 

.06 

.43 

-.02 

-.01 

.19 

.13 

13.  Decentralization 

.41 

.36 

.29 

-.04 

.38 

.24 

19.  Leader:  Task  Orientation 

-.18 

-.06 

.01 

-.04 

.04 

.09 

21.  Leader:  Problem  Solving 

-.18 

-.07 

-.05 

-.10 

.05 

.10 

22.  No.  Supervisors 

-.39 

.01 

.14 

.11 

.19 

.16 

23.  No.  Teams 

-.13 

.38 

.46 

.23 

.42 

.21 

24.  No.  Reorganizations 

-.16 

.17 

.13 

.02 

.07 

.07 

25.  No.  Changes  in  Job  Duties 

-.07 

-.02 

.11 

-.15 

.04 

.06 

26.  % Time  Spent  in  Teams 

-.05 

.01 

.64 

-.09 

.42 

.19 

27.  No.  Training  Topics 

.10 

.41 

-.02 

.16 

.20 

.16 

28.  No.  Training  Methods 

.04 

.21 

.03 

.34 

.16 

.15 

29.  No.  Selection  Methods 

.00 

.00 

-.02 

.59 

.35 

.14 

30.  No.  Recruitment  Sources 

.10 

.06 

.16 

.01 

.04 

.08 

3 1 . No.  Compensation  Elements  .23 

.19 

.29 

.04 

.17 

.16 

32.  No.  Benefits  Elements 

.59 

-.10 

-.08 

.22 

.42 

.24 

33.  Job  Rotation  Practices 

-.14 

-.09 

.33 

' .08 

.15 

.12 

Rc 

.66 

.57 

.53 

.49 

Percent  of  Variance 

21 

13 

10 

8 

Eigenvalue 

.78 

.49 

.38 

.32 

Note.  Statistics  are  based  on  30  occupations  with  Organizational  Context  questionnaire  responses 
from  at  least  4 incumbents  (mean  number  of  incumbents  = 18.47,  median  = 12,  harmonic  mean 
= 10.14).  FI  = Decision  Making  Authority,  F2  = Goal  Setting,  F3  = Team  Structure,  F4  = Skill 
Variety. 

LF  = Sum  of  squared  rotated  standardized  discriminant  function  coefficients  across  4 functions. 
T|2  = Variance  in  Organizational  Context  ratings  accounted  by  cells. 

The  statistics  “Re”,  “Percent  of  Variance”,  and  “Eigenvalue”,  were  calculated  based  on  the 
unrotated  discriminant  functions. 
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Table  8- 13b 

Rotated  Correlations  Between  Discriminating  Variables  and  Canonical 
Discriminant  Functions  Across  Job  within  Organization  Cells:  Organizational 
Context 


Functions 


Descriptor/Scale 

FI 

F2 

F3 

F4 

F5 

IF2 

Tl2 

1.  Empowerment 

.28 

.18 

.33 

.05 

-.07 

.23 

.37 

2.  Autonomy 

.16 

-.15 

.14 

.27 

.11 

.15 

.36 

4.  Skill  Variety 

.13 

.02 

.03 

.06 

.37 

.16 

.35 

5.  Task  Significance 

-.04 

-.02 

.06 

.17 

-.05 

.04 

.30 

7.  Role  Conflict 

.06 

.06 

.11 

.02 

.01 

.02 

.24 

8.  Role  Overload 

.36 

.18 

.05 

-.07 

-.08 

.17 

.30 

9.  Adequate  Resources 

.01 

.01 

-.17 

.00 

.00 

.03 

.28 

10.  Role  Negotiability 

.25 

.09 

-.03 

-.08 

-.01 

-.08 

.24 

1 1 . Goal  Specificity 

.10 

-.02 

.50 

.13 

.01 

.28 

.42 

12.  Goal  Feedback 

-.01 

.08 

.45 

-.13 

-.12 

.24 

.36 

13.  Decentralization 

.34 

.02 

.22 

.01 

-.01 

.16 

.37 

19.  Leader:  Task  Orientation 

-.16 

.21 

.04 

.08 

.00 

.08 

.35 

20.  Leader:  Problem  solving 

-.04 

.30 

-.10 

.17 

.16 

.16 

.33 

22.  No.  Supervisors 

-.13 

.18 

-.19 

.46 

.08 

.31 

.42 

23.  No.  Teams 

-.05 

-.02 

.25 

.18 

.08 

.11 

.31 

24.  No.  Reorganizations 

.11 

.46 

.10 

.01 

.12 

.25 

.36 

25.  No.  Changes  in  Job  Duties 

.05 

.39 

.02 

-.14 

-.07 

.18 

.31 

26.  % Time  Spent  in  Teams 

.13 

.34 

.14 

.09 

-.07 

.16 

.31 

27.  No.  Training  Topics 

.09 

.15 

.53 

.12 

.22 

.38 

.49 

28.  No.  Training  Methods 

.00 

.21 

.50 

-.14 

.37 

.46 

.48 

29.  No.  Selection  Methods 

-.17 

.02 

.13 

-.08 

.51 

.31 

.40 

30.  No.  Recruitment  Sources 

.12 

.17 

.07 

.09 

.14 

.08 

.29 

3 1 . No.  Compensation  Elements  .39 

.06 

.25 

.19 

.06 

.26 

.41 

32.  No.  Benefits  Elements 

.48 

-.22 

-.12 

-.11 

.16 

.33 

.49 

33.  Job  Rotation  Practices 

.06 

.18 

.16 

-.04 

-.21 

.11 

.29 

Sc 

.85 

.79 

.77 

.74 

.70 

Percent  of  Variance 

17 

11 

10 

8 

7 

Eigenvalue 

2.50 

1.69 

1.42 

1.22 

.97 

Note.  Statistics  are  based  on  50  job  within  organization  cells  with  Organizational 
Context  questionnaire  responses  from  at  least  4 incumbents  (mean  number  of 
incumbents  = 4.85,  median  = 4.00,  harmonic  mean  = 4.72).  FI  = Decision  Making 
Authority,  F2  = Organizational  Change,  F3  = Goal  Setting,  F4  = Change  in 
Supervision,  F5  = Skill  Variety. 

EF  = Sum  of  squared  rotated  standardized  discriminant  function  coefficients  across 
5 functions. 

r\  = Variance  in  Organizational  Context  ratings  accounted  by  cells. 

The  statistics  “Re”,  “Percent  of  Variance”,  and  “Eigenvalue”,  were  calculated  based 
on  the  unrotated  discriminant  functions. 
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Table  8-17 

Correlations  Among  Group  Mean  Ratings  Profiles  Between  and  Within  Three  Example 
Occupations:  Organizational  Context 


Occ.  1 
Group  1 
n=25 

Occ.  1 
Group  2 
n=25 

Occ.  2 
Group  1 
n=32 

Occ.  2 
Group  2 
n=32 

Occ.  3 
Group  1 
n=34 

Occ.  3 
Group  2 
n=34 

Occ.  1 Group  1 

1.00 

Occ.  1 Group  2 

0.70 

1.00 

Occ.  1 Group  1 

-0.51 

-0.58 

1.00 

Occ.  1 Group  2 

-0.44 

-0.27 

0.14 

1.00 

Occ.  1 Group  1 

-0.29 

-0.25 

-0.31 

-0.22 

1.00 

Occ.  1 Group  2 

-0.57 

-0.61 

0.19 

-0.10 

0.14 

1.00 

Note.  N=33.  Correlation  coefficients  greater  than  10.321  were  significant  at  the  g<.05  level. 

Occ.  l=First  line  Supervisors  and  Managers/Supervisors,  Clerical  and  Administrative  Workers, 
Occ.  2=Secretaries  Except  Legal  and  Medical,  Occ.  3=General  Office  Clerks. 


Table  8- 18a 

Descriptive  Statistics  for  Organizational  Representative  (CATD  Data:  Orpaniyatinnal 
Context 


Item  # (s) 

Item/Scale  Label 

N 

M 

SD 

6 

No.  of  Org.  Locations 

661 

177.24 

527.88 

7 

No.  of  Countries  do  Business 

661 

12.37 

40.17 

8 

Full  Time  Org.  Employees 

661 

13703.37 

51304.79 

9&10 

Total  Employees  (Establishment) 

661 

322.71 

659.21 

11 

No.  New  Employees  Joined  Last  Year 

620 

55.77 

120.25 

12 

No.  Employees  Last  Year 

630 

298.58 

601.02 

18 

No.  Management  Levels 

605 

10.18 

38.50 

22 

No.  Job  Titles 

661 

57.55 

117.37 

24 

No.  New  Jobs  Created  (Last  5 Yrs) 

553 

10.04 

25.15 

31 

Information  Sharing  (%  Informed) 

659 

49.69 

29.52 

34 

% Working  in  Teams 

661 

31.78 

36.57 

35a 

% in  Functional  Teams 

661 

29.25 

34.71 

35b 

% in  Cross-Functional  Teams 

661 

19.21 

28.40 

35c 

% in  Management  Teams 

661 

8.41 

17.59 

35d 

% in  Project  Teams 

661 

7.37 

16.86 

35c 

% in  Quality  Improve  Teams 

661 

12.90 

24.22 

39 

% Managers  Set  Goals 

649 

70.05 

41.60 

40 

% Managers  Negotiate  Goals 

640 

64.97 

44.02 

41 

% Non-Managers  Set  Goals 

650 

56.79 

44.58 

42 

% Non-Managers  Negotiate  Goals 

647 

44.09 

45.28 

43 

No.  Performance  Reviews  in  Last  2 Yr 

652 

2.17 

1.50 

49a 

% Attend  No  Training 

523 

20.59 

29.71 

49b 

% Attend  One  Training  Courses 

522 

24.47 

28.75 

49c 

% Attend  Two+  Training  Courses 

640 

40.16 

38.17 

50d 

% Get  Quality  Control  Training 

642 

25.06 

38.33 

66 

% Pay  Adjusted  on  Evaluate  Studies 

638 

50.18 

44.97 

67 

% Pay  Adjusted  on  Comparisons 

642 

54.93 

43.53 

68a 

% w / Salary 

650 

52.17 

40.16 

68b 

% w / Profit-Sharing 

655 

25.57 

40.34 

68c 

% w / Skill-Based  Pay 

642 

26.62 

38.70 

68d 

% w/  Indiv  Performance-Based  Pay 

650 

52.29 

44.46 

68e 

% w/  Team  Performance-Based  Pay 

649 

12.50 

29.10 

68f 

% w/  Custm  Satisfaction-Based  Pay 

647 

17.45 

34.70 

68g 

% w/  Seniority-Based  Pay 

646 

31.08 

41.45 

68h 

% w/  Hay  Points 

638 

13.91 

30.51 

69a 

% w/  Stocks 

654 

13.97 

31.36 

69b 

% w/  Retirement  Plan 

657 

73.99 

39.46 

69c 

% w/  Medical  Insurance 

659 

81.00 

32.06 

69d 

% w/  Life  Insurance 

656 

76.18 

37.16 

Table  8- 18a  (continued) 

Descriptive  Statistics  for  Organizational  Representative  (CATD  Data:  Organizational 
Context 


Item  # (s) 

Item/Scale  Label 

N 

M 

SD 

69e 

% w/  Disability  Insurance 

653 

70.21 

41.48 

69f 

% w/  Flex  Hours 

651 

35.44 

41.28 

69g 

% w/  Daycare 

659 

7.77 

25.88 

69h 

% w/  Paid  Vacation 

660 

84.90 

28.93 
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Table  8- 18b 


Organizational  Context 

Item  # Item/scale  Label  (#  items) 

Rating  Scale 

N 

M 

SD 

14 

Use  of  Contractors:  Extent 

l=not  at  all 
5=to  a great  extent 

658 

2.25 

1.10 

15 

Use  of  Contractors:  Freq 

1=  never 
5=always 

657 

2.66 

1.19 

20 

No.  Formal  Documents 

# checked  out  of  7 

661 

5.72 

1.34 

21 

Document  Relate  to  Perform 
(5) 

l=not  at  all 

5=  to  a great  extent 

585 

3.81 

0.88 

23 

Changes  in  Job  Duties:  Freq 

1=  never 
5=always 

656 

2.69 

0.90 

25 

No.  Reorganizations 

0=never,  1=1,  2=2, 
3=3, 4=4,  5=5, 

6=6  or  more 

562 

0.91 

1.26 

27 

No.  Org  Chart  Revisions 

0=never,  1=1,  2=2, 
3=3, 4=4,  5=5  or  more 

266 

2.96 

1.88 

28 

No.  Specializations 

# checked  out  of  15 

661 

10.76 

3.02 

29 

Standardization  (14) 

l=not  standardized 
5=completely  standardized 

661 

3.84 

0.68 

30 

Decentralization  (5) 

l=not  at  all 

5=  to  a great  extent 

658 

2.33 

0.84 

32 

Extent  Use  of  Teams 

l=not  at  all 

5=  to  a great  extent 

661 

3.03 

1.49 

33 

Team  Accountability 

l=not  at  all 

5=  to  a great  extent 

661 

3.03 

1.50 

37 

Dept  Heads  Set  Quant  Goals 

l=none 

5=all 

655 

3.29 

1.63 

45 

No.  Training  Methods 

# checked  out  of  1 1 

659 

6.56 

2.61 

46 

Trg  Based  on  Needs  Analysis 

l=few 

5=all 

648 

2.76 

1.42 

47 

Freq  of  Training  Evaluation 

l=never 

5=always 

650 

3.04 

1.38 

48 

No.  Training  Topics 

# checked  out  of  7 

658 

4.47 

1.92 

55 

Job  Rotation  Policies 

l=no  job  rotation 
2=rotate  within  workgroup 
3=rotate  across  workgroups 
4=rotate  across  departments 

654 

2.48 

1.20 

57 

Types  of  Recruit  Data  Collected  # checked  out  of  6 

654 

1.13 

1.82 

62 

Select  Systems  Based  on  Job 
Analysis 

l=none 

5=all 

653 

2.94 

1.49 

ERIC 


Table  8- 18b  (continued) 

Descriptive  Statistics  and  Rating  Scales  Used  for  Organizational  Representative  (CATD  Data: 
Organizational  Context 


Item  # Item/scale  Label  (#  items) 

Rating  Scale 

N 

M 

SD 

63 

Select  Systems  Validated 

l=none 

5=all 

649 

2.86 

1.43 

64 

Spend  Money  on  Data 

l=not  at  all 

5=  to  a great  extent 

634  J 

2.36 

1.18 

65 

Use  Data  for  Org.  Decisions 

l=not  at  all 

5=  to  a great  extent 

642 

2.93 

1.21 

874 

o 

ERIC 


Table  8- 18c 

Descriptive  Statistics  for  Organizational  Representative  (CATP  Data: 
Organizational  Values 


Item  # 

Organizational  Value3 

N 

M 

SD 

70a 

Employment  Security 

653 

5.43 

1.47 

70b 

Risk  Taking 

645 

3.42 

1.75 

70c 

Flexibility 

654 

5.06 

1.52 

70d 

Analytical  Oriented 

646 

4.05 

1.75 

70e 

People  Oriented 

657 

5.87 

1.32 

70f 

Fairness 

656 

6.10 

1.07 

70g 

Competitiveness 

656 

5.07 

1.69 

70h 

Collaboration 

650 

5.28 

1.44 

70i 

Adaptability 

656 

5.52 

1.26 

70j 

Predictability 

653 

4.87 

1.44 

70k 

Innovation 

656 

4.95 

1.45 

701 

Social  responsibility 

653 

5.41 

1.52 

70m 

Quality 

657 

6.23 

0.99 

70n 

Results  oriented 

653 

5.76 

1.29 

70o 

Tolerance 

654 

5.52 

1.24 

70p 

Taking  Advantage  of  Opportunity  655 

5.42 

1.28 

70q 

Customer  Oriented 

655 

6.06 

1.37 

70r 

Action  Oriented 

648 

5.47 

1.32 

70s 

Stability 

655 

5.52 

1.36 

70t 

Autonomy 

646 

4.75 

1.49 

70u 

Attention  to  Details 

656 

5.68 

1.15 

70v 

Team  Oriented 

657 

5.27 

1.51 

70w 

Sharing  Information  Freely 

656 

5.33 

1.41 

70x 

Willing  to  Experiment 

655 

4.64 

1.52 

70y 

Aggressiveness 

654 

4.90 

1.43 

70z 

Precision 

653 

5.37 

1.22 

70aa 

Achievement  Oriented 

656 

5.53 

1.30 

70bb 

Supportiveness 

657 

5.73 

1.17 

“The  values  were  rated  on  a 7-point  Likert-type  scale  where  1 = least 
characteristic  and  7 = most  characteristic. 
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Table  8-18d 

Descriptive  Statistics  for  Organizational  Representative  fCATP  Data: 
Dichotomous  Oreanizational  Context  Variables 


Percentage 


Item# 

Item  Label 

N 

Yes 

No 

13 

Downsized  in  Last  5 Years 

567 

74.5 

25.5 

38 

Publicize  One  Quantitative  Goal 

645 

65.4 

34.6 

44 

Formal  Training 

660 

82.1 

17.9 

45a 

Training  Method  1:  Case  Studies 

528 

52.1 

47.9 

45b 

Trg  Method  2:  Conference/Discuss 

537 

90.1 

9.9 

45c 

Trg  Method  3:  Lectures 

537 

84.4 

15.6 

45d 

Trg  Method  4:  Business  Games 

527 

30.6 

69.4 

45e 

Trg  Method  5:  Simulators 

537 

20.3 

79.7 

45f 

Trg  Method  6:  Films 

540 

88.7 

11.3 

45g 

Trg  Method  7:  Interact  Videos 

530 

40.2 

59.8 

45h 

Trg  Method  8:  Workbooks 

538 

74.9 

25.1 

45i 

Trg  Method  9:  Role-Play 

536 

67.4 

32.6 

45j 

Trg  Method  10:  Comput  Inst 

537 

61.5 

38.5 

45k 

Trg  Method  1 1 : Audio  Cassettes 

534 

53.0 

47.0 

48a 

Diversity  Training 

532 

54.9 

45.1 

48b 

Team  Skills  Training 

536 

70.3 

29.7 

48c 

Quality  Control  Training 

530 

68.7 

31.3 

48d 

Basic  Business  Training 

535 

• 32.5 

67.5 

48e 

Problem  Solving  Training 

534 

70.6 

29.4 

48f 

Leadership  Training 

538 

75.5 

24.5 

48g 

Customer  Service  Training 

538 

78.4 

21.6 

53 

Continuous  Learning  Programs 

657 

67.1 

32.9 

54 

Financial  Assist  for  Training 

660 

70.0 

30.0 

56 

Formal  Recruitment  Plan 

656 

43.0 

57.0 

58 

Realistic  Job  Previews 

654 

39.1 

60.9 

59 

Formal  Orientation  Programs 

659 

64.8 

35.2 

60 

Formal  Mentoring  Programs 

655 

28.8 

74.2 

61 

Formal  Selection  Systems 

657 

83.7 

16.3 

876 


Table  8- 19a 

Principal  Factor  Analysis  Pattern  Matrix  for  Organizational  Representative  fCATT)  Data: 
Organizational  Structure 


Factor 


Item  # Item  Label 

FI 

F2 

F3 

F4 

F5 

F6  Communality 

6 

No.  Locations 

-.01 

.03 

.00 

.03 

.73 

.09 

.54 

7 

No.  Countries  do  Business 

.12 

-.10 

.04 

.01 

.26 

.24 

'.15 

8 

Full  Time  Org.  Employees 

.01 

.02 

.04 

.13 

.73 

.07 

.55 

9+10 

Total  Employees  (Establishment) 

-.08 

.16 

.05 

.05 

.05 

.62 

.42 

18 

No.  Management  Levels 

.05 

.14 

-.03 

.07 

.24 

-.05 

.09 

20 

No.  Formal  Documents 

.12 

.68 

.12 

.01 

-.01 

.28 

.57 

21a 

Formal  1 : Employ  Contracts 

.17 

.20 

.19 

-.05 

-.07 

.12 

.12 

21b 

Formal  2:  Org.  Chart 

.06 

.43 

.07 

.07 

-.04 

.15 

.22 

21c 

Formal  3:  Job  Descriptions 

.04 

.56 

.05 

.07 

.04 

-.03 

.33 

21d 

Formal  4:  Procedure  Manuals 

.10 

.78 

.02 

.05 

.06 

-.01 

.62 

21e 

Formal  5:  Policy  Manuals 

.02 

.81 

.00 

.03 

.10 

.08 

.67 

22 

No.  Job  Titles  (Establishment) 

-.11 

.12 

.05 

.06 

.07 

.59 

.38 

28 

No.  Specializations  (Establishment)  .08 

.10 

.23 

-.10 

-.04 

.27 

.16 

29 

Standardization 

.19 

.29 

.19 

.09 

.06 

-.02 

.17 

30a 

Decentral  1:  Monitor  Quality 

.09 

.06 

.63 

.07 

.10 

.20 

.46 

30b 

Decentral  2:  Work  Flow 

.05 

.11 

.66 

.11 

.08 

-.03 

.46 

30c 

Decentral  3:  New  Equipment 

.13 

.06 

.45 

.16 

-.07 

.00 

.25 

30d 

Decentral  4:  New  Products 

.16 

.09 

.57 

.21 

-.08 

.03 

.42 

30e 

Decentral  5:  New  Members 

.20 

-.01 

.46 

.12 

.04 

.07 

.28 

31a 

Info  Sharing  1 : Finance 

.17 

.13 

.00 

.69 

.10 

.26 

.60 

31b 

Info  Sharing  2:  Unit  Finance 

.20 

.07 

.04 

.68 

.07 

.23 

.57 

31c 

Info  Sharing  3:  New  Tech. 

.12 

.06 

.21 

.46 

.05 

-.11 

.29 

31d 

Info  Sharing  4:  Bus.  Plans 

.03 

.13 

.26 

.60 

.03 

-.12 

.46 

31e 

Info  Sharing  5:  Competitors 

.08 

-.05 

.25 

.48 

.11 

-.07 

.32 

32 

Extent  Use  of  Teams 

.76 

.15 

.22 

.06 

.07 

-.14 

.67 

33 

Team  Accountability 

.73 

.20 

.19 

.09 

.10 

.05 

.64 

34 

% Working  in  Teams 

.76 

.07 

.08 

.05 

.10 

-.13 

.62 

35a 

% in  Functional  Teams 

.73 

.03 

.08 

.07 

.14 

-.13 

.58 

35b 

% in  Cross-Functional  Teams 

.62 

.09 

.12 

.07 

.01 

-.09 

.42 

35c 

% in  Management  Teams 

.40 

.07 

-.03 

.14 

-.15 

.04 

.21 

35d 

% in  Project  Teams 

.38 

.01 

.28 

.22 

.02 

.11 

.28 

35e 

% in  Quality  Improve  Teams 

.38 

.00 

.24 

.08 

-.01 

.03 

.21 

Percent  of  Variance 

11 

8 

7 

6 

4 

4 

Eigenvalue 

3.38 

2.58 

2.15 

2.01 

1.34 

1.27 

Note.  N=245.  Factor  labels:  Fl=Use  of  Teams,  F2=Formalization/Standardization, 
F3=Information  Sharing,  F4=Decentralization,  F5=Establishment  size/Specialization, 
F6=Organization  Size. 
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Table  8-19b 

Principal  Factor  Analysis  Pattern  Matrix  for  Organizational  Representative  (CATP  Data: 
HR  Practices 


Factor 


Item#  Item  Label 

FI 

F2 

F3 

F4  Communality 

36 

Formal  mission  statement 

.46 

.37 

.17 

.05 

.37 

37 

Dept  heads  set  quant  goals 

.29 

.21 

.52 

.05 

.40 

38 

Publicize  one  quantitative  goal 

.23 

.22 

.53 

.19 

.42 

39 

% managers  set  goals 

.17 

.11 

.72 

.17 

.59 

40 

% managers  negotiate  goals 

.08 

.14 

.74 

.06 

.58 

41 

% non-managers  set  goals 

.23 

.10 

.58 

.34 

.51 

42 

% non-managers  negotiate  goals 

.13 

.01 

.62 

.13 

.42 

43 

No.  performance  reviews  in  2 years 

.17 

-.01 

.21 

.14 

.09 

45a 

Trg  method  1:  Case  studies 

.56 

.27 

.13 

-.02 

.41 

45b 

Trg  method  2:  Conf/group 

.78 

.12 

.18 

-.01 

.66 

45c 

Trg  method  3:  Lectures 

.76 

.14 

.17 

-.05 

.64 

45d 

Trg  method  4:  Business  games 

.41 

.04 

.06 

.28 

.25 

45e 

Trg  method  5:  Simulators 

.26 

.05 

.04 

.11 

.08 

45f 

Trg  method  6:  Film/video 

.79 

.21 

.09 

-.01 

.68 

45g 

Trg  method  7:  Interact  videos 

.49 

.14 

.07 

.10 

.28 

45h 

Trg  method  8:  Workbooks 

.66 

.16 

.12 

.10 

.48 

45i 

Trg  method  9:  Role-play 

.73 

.10 

.17 

.01 

.57 

45j 

Trg  method  10:  Comput  Inst 

.53 

.19 

.20 

.12 

.38 

45k 

Trg  method  11:  Audio  Cassettes 

.49 

.25 

.15 

-.07 

.33 

46 

Trg  based  on  needs  analys 

.73 

.02 

.20 

.09 

.57 

47 

Freq.  of  training  evaluation 

.79 

.10 

.15 

.00 

.66 

48a 

Diversity  training 

.57 

.18 

.18 

.06 

.39 

48b 

Team  training 

.71 

.08 

.19 

.11 

.56 

48c 

Quality  control  training 

.64 

.01 

.10 

.07 

.43 

48d 

Basic  business  training 

.38 

.04 

.08 

.19 

.19 

48e 

Problem  solving  training 

.75 

.09 

.11 

.04 

.59 

48f 

Leadership  training 

.79 

.15 

.11 

-.03 

.66 

48g 

Customer  service  training 

.64 

-.02 

.14 

.12 

.44 

49b 

% attend  one  training  course 

.24 

.22 

.21 

.04 

.15 

49c 

% attend  2+  training  courses 

.66 

.02 

.00 

-.08 

.44 

50a 

% get  quality  control  training 

.42 

.22 

.06 

.16 

.26 

53 

Continuous  learning  programs 

.26 

.12 

.12 

.31 

.19 

54 

Financial  assist  for  training 

.24 

.38 

.16 

-.02 

.23 

55 

Job  rotation  policies 

.27 

.06 

.14 

.43 

.28 

57 

Types  of  recruit  data  collected 

.41 

.13 

.14 

.30 

.30 

58 

Realistic  job  previews 

.50 

.11 

.13 

.21 

.32 

62 

Select  systems  based  on  job  analysis  .43 

.10 

.06 

.25 

.26 

63 

Select  systems  validated 

.45 

.01 

.09 

.28 

.29 

66 

% pay  adjusted  on  evaluate  studies 

.16 

.11 

.10 

.46 

.26 
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Table  8- 19b  (continued) 


HR  Practices 
Item  # Item  Label 

FI 

F2 

Factor 

F3  F4  Communality 

67 

% Pay  Adjusted  on  Comparisons 

.27 

.25 

.08 

.29 

.23 

68a 

% w/  Salary 

.05 

.41 

.05 

.00 

.18 

68b 

% w/  Profit  Sharing 

.02 

.31 

.07 

.36 

.23 

68c 

% w/  Skill-Based  Pay 

.03 

.01 

.05 

.40 

.16 

68d 

% w/  Indiv  Perform-Based  Pay 

-.05 

.11 

.00 

.52 

.28 

68e 

% w/  Team  Perform-Based  Pay 

-.04 

.03 

.08 

.46 

.22 

68f 

% w/  Custm  Satis-Based  Pay 

-.02 

-.15 

.10 

.58 

.37 

68g 

% w/  Seniority-Based  Pay 

.08 

-.01 

.07 

-.05 

.01 

68h 

% w/  Hay  Points 

.07 

.22 

.07 

.17 

.08 

69a 

% w/  Stock  Options 

.11 

.18 

.04 

.25 

.11 

69b 

% w/  Retirement  Plan 

.16 

.76 

.11 

.01 

.62 

69c 

% w/  Medical  Insurance 

.10 

.83 

.03 

.04 

.69 

69d 

% w/  Life  Insurance 

.11 

.84 

-.01 

.06 

.73 

69e 

% w/  Disability  Insurance 

.19 

.64 

.07 

.09 

.46 

69f 

% w/  Flex  Time 

.02 

-.22 

-.02 

.33 

.16 

69g 

% w/  Day  Care 

.14 

.03 

.16 

.01 

.04 

69h 

% w/  Paid  Vacation 

.11 

.61 

.08 

.08 

.39 

Percent  of  Variance 

19 

8 

6 

5 

Eigenvalue 

10.63 

4.22 

3.06 

2.66 

Setting,  F4=High  Performance  HR  Practices. 
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Table  8- 19c 


Organizational  Values 
Item  # Item  Label 

FI 

F2 

Factor 

F3 

F4  Communality 

70a 

Security  of  Employment 

.07 

.05 

.08 

.54 

.31 

70b 

Risk-Taking 

.04 

-.10 

.42 

-.05 

.19 

70c 

Flexibility 

.56 

-.07 

.20 

.10 

.37 

70d 

Analytical  Orientation 

.08 

.22 

.47 

.02 

.27 

70e 

People  Orientation 

.61 

.18 

.11 

.04 

.42 

70f 

Fairness 

.49 

.30 

.11 

.25 

.40 

70g 

Competitiveness 

.05 

.16 

.51 

-.03 

.29 

70h 

Collaboration 

.54 

.16 

.18 

.09 

.36 

70i 

Adaptability 

.60 

.18 

.16 

.18 

.45 

70j 

Predictability 

.09 

.19 

.00 

.54 

.34 

70k 

Innovation 

.34 

.26 

.47 

.11 

.42 

701 

Social  Responsibility 

.40 

.31 

.14 

.27 

.35 

70m 

Quality 

.44 

.53 

.09 

.21 

.53 

70n 

Results  Orientation 

.25 

.57 

.32 

.14 

.51 

70o 

Tolerance 

.56 

.32 

-.04 

.22 

.47 

70p 

Taking  Advantage  of  Opportunity 

.30 

.39 

.52 

.07 

.52 

70q 

Customer  Service  Orientation 

.43 

.19 

.12 

-.12 

.25 

70r 

Action  Orientation 

.32 

.53 

.33 

-.01 

.48 

70s 

Stability 

.29 

.20 

-.02 

.66 

.56 

70t 

Autonomy 

.25 

.18 

.34 

.36 

.34 

70u 

Attention  to  Detail 

.25 

.63 

.06 

.28 

.54 

70v 

Team  Orientation 

.50 

.30 

.13 

.22 

.41 

70w 

Sharing  Information  Freely 

.54 

.16 

.15 

.31 

.44 

70x 

Willingness  to  Experiment 

.37 

.01 

.49 

.17 

.41 

70y 

Aggressiveness 

.09 

.17 

.59 

.16 

.40 

70z 

Precision 

.20 

.60 

.14 

.35 

.55 

70aa 

Achievement  Orientation 

.17 

.59 

.38 

.25 

.59 

70bb 

Supportiveness 

.60 

.34 

-.02 

.43 

.66 

Percent  of  Variance  1 5 

11 

9 

8 

Eigenvalue 

4.15 

3.07 

2.45 

2.14 

Note.  N = 295.  Factor  labels:  FI  = People  Orientation,  F2  = Risk  Taking,  F3  = Attention  to 
Detail,  F4  = Stability. 
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Table  8-20 

Factor-Based  Composite  Intercorrelations  for  Organizational  Representative  (CATO  Data:  Organizational  Context 

Composite  Label  1 2 3 4 5 6 7 8 9 10  11  12  13 

1.  UseofTeams 
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Table  8-21 

Principal  Factor  Analysis  Pattern  Matrix  for  Organizational  Representative 
(CATO  Data:  Factor-Based  Composites 


Factor 


Composite  Label 

FI 

F2 

F3 

Communality 

People  Orientation 

.74 

.25 

.00 

.61 

Attention  to  Detail 

.73 

.19 

.15 

.59 

Risk-Taking 

.62 

.36 

.06 

.51 

Stability 

.55 

-.06 

-.03 

.31 

Goal  Setting 

.15 

.54 

.27 

.39 

Information  Sharing 

.09 

.51 

.32 

.37 

Use  of  Teams 

.22 

.46 

.22 

.31 

High  Performance  HR  Practices 

.15 

.46 

-.07 

.23 

Multiple  Skill  Training 

.20 

.19 

.58 

.41 

Formalization/Standardization 

.06 

.11 

.50 

.27 

Employee  Benefits 

.01 

.24 

.43 

.25 

Establishment  Size 

-.05 

-.08 

.41 

.18 

Decentralization 

.06 

.33 

.38 

.26 

Organization  Size 

-.04 

.15 

.19 

.06 

Percent  of  Variance 

13.71 

10.36 

9.79 

Eigenvalue 

1.92 

1.45 

.1.37 

Note.  N = 288.  Factor  labels:  FI  = Organizational  Values,  F2  = High- 
Performance  Practices,  F3  = Establishment  Size. 
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Table  8-22 

Factor  Loadings  for  Second-Order  Confirmatory  Factor  Analysis:  Organizational  Context 

Factor 

Item  #(s)  Indicator  Label  FI  F2  F3  F4  F5  F6 


32 

Extent  Use  of  Teams  .96 

33 

T earn  Accountability  1 .00 

34 

% Working  in  Teams  .85 

30a 

Decentral  1:  Monitor  Quality  .89 

30b 

Decentral  2:  Work  Flow  1.00 

30c 

Decentral  3:  New  Equipment  .90 

30d 

Decentral  4:  New  Products  .95 

3 la&3  lb 

Info  Sharing  1 : Finance 

31c 

Info  Sharing  3:  New  Tech 

.71 

3 Id 

Info  Sharing  4:  Bus.  Plans 

.68 

31e 

Info  Sharing  5:  Competitors 

.68 

70b&70k&70x 

Risk-Taking  Values  1:  Innovation 

1.00 

70g&70p&70q& 

Risk-Taking  Values  2:  Competitiveness 

.70 

70r&70y&70aa 

70c&70t 

Risk-Taking  Values  1:  Autonomy 

.59 

46&47 

Use  Data  to  Develop  & Evaluate  Trg 

1.00 

62&63 

Use  Data  to  Develop  Selection  Sys 

.87 

64&65 

Collect/Use  Data  for  Org  Decisions 

.67 

55 

Job  Rotation  Policies 

.95 

48 

No.  Training  Topics 

1.00 

Note.  N = 326.  Factor  labels:  FI  = Use  of  Teams,  F2  = Decentralization,  F3  = Information 

Sharing,  F4  = Risk-taking  Values,  F5  = Use  Data  in  Decision  Making,  F6  = Multiple  Skill 

Training. 
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Table  8-23 

Adjusted  Means  of  Factor-Based  Composites  by  Industry  Type:  Organizational  Context 
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Table  8-24a 

Principal  Components  Analysis  Pattern  Matrix  of  Tnr.nmhent  Data: 
Organizational  Structure 


Factor 


Descriptor/Scale 

FI 

F2 

F3 

Communality 

1.  Empowerment 

.78 

.00 

.11 

.61 

2.  Autonomy 

.78 

.01 

-.07 

.61 

3.  Skill  Variety 

.72 

-.10 

.21 

.57 

4.  Task  Significance 

.42 

-.17 

.16 

.23 

5.  Decentralization 

.68 

-.15 

.01 

.49 

6.  Leader:  Task  Oriented 

-.10 

.80 

-.11 

.67 

7.  Leader:  Problem  Solving 

-.11 

.80 

.20 

.69 

8.  No.  Teams 

.25 

-.04 

.60 

.42 

9.  No.  Supervisors 

-.27 

.10 

.55 

.38 

10.  No.  Reorganizations 

.05 

.07 

.51 

.27 

1L%  Time  Spent  in  Teams 

.21 

-.08 

.36 

:i8 

Percent  of  Variance 

23.27 

12.55 

10.82 

Eigenvalue 

2.56 

1.38 

1.19 

Note.  N = 30.  The  correlation  matrix  was  based  on  means  calculated  at  the 
organization  level. 

FI  = Decentralization\Employee  Empowerment;  F2  = Task-Oriented 
Leadership;  F3  = Work  in  Teams.  These  loadings  are  based  on  orthogonal 
varimax  rotation. 


Table  8-24b 


Resources  (HR)  Practices 
Descriptor/Scale 

Factor 

FI 

F2 

F3  Communality 

1.  No.  Training  Topics 

.62 

.18 

.15 

.44 

2.  No.  Training  Methods 

.69 

.12 

.22 

.55 

3.  No.  Selection  Methods 

.49 

.00 

.00 

.24 

4.  No.  Recruitment  Sources 

.49 

.10 

.15 

.27 

5.  No.  Compensation  Elements  .54 

.04 

.21 

.34 

6.  No.  Benefits  Elements 

.52 

.07 

.29 

.36 

7.  Job  Rotation  Policies 

.34 

.30 

-.09 

.22 

8.  Role  Conflict 

.12 

.74 

.15 

.59 

9.  Changes  in  Job  Duties 

.06 

.74 

.05 

.56 

10.  Role  Overload 

.20 

.58 

.22 

.42 

11.  Adequate  Resources 

.04 

-.49 

-.33 

.35 

12.  Role  Negotiability 

.16 

.30 

.37 

.25 

13.  Goal  Specificity 

.24 

.19 

.73 

.63 

14.  Goal  Feedback 

.25 

.14 

.72 

.60 

Percent  of  Variance 

14.64 

12.50 

12.29 

Eigenvalue 

2.05 

1.75 

1.72 

Note.  N = 30.  The  correlation  matrix  was  based  on  means  calculated  at  the 
organization  level. 

FI  = HR  Systems;  F2  = Role  Conflict;  F3  = Individual  Goal  Setting.  These 
loadings  are  based  on  orthogonal  varimax  rotation. 


388 


Chapter  9 
Abilities: 


Evidence  for  the  Reliability  and  Validity  of  the  Measures 

Edwin  A.  Fleishman 
David  P.  Costanza 
Joanne  C.  Marshall-Mies 
Management  Research  Institute,  Incorporated 


The  term  ability  has  been  assigned  a variety  of  meanings.  Fleishman  (1972a,  1972b, 
1975a,  1975b,  1982)  defines  abilities  as  relatively  enduring  attributes  of  an  individual's  capability 
for  performing  a particular  range  of  different  tasks.  Abilities  are  regarded  as  traits  in  that  they 
exhibit  some  degree  of  stability  over  relatively  long  periods  of  time.  It  is  recognized,  however, 
that  abilities  may  develop  over  time  and  with  exposure  to  multiple  situations  (Snow  & Lohman, 
1984).  Additionally,  there  is  a distinction  between  abilities  and  skills  (Bilodeau,  1966; 

Fleishman,  1966, 1972a).  An  ability  is  a general  trait  of  an  individual  that  is  based  on 
relationships  among  performances  of  individuals  observed  across  a range  of  different  tasks. 

As  jobs  are  often  described  in  terms  of  the  tasks  required  to  perform  them,  the 
identification  and  analysis  of  job  tasks  and  of  the  abilities  that  are  relevant  to  those  tasks  can 
provide  essential  information  in  describing  and  understanding  various  occupations.  In  the 
remainder  of  this  chapter,  we  first  briefly  discuss  the  nature  and  characteristics  of  job-relevant 
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abilities,  as  well  as  how  the  abilities  taxonomy  was  developed  and  evaluated  in  the  present  effort. 
Second,  we  present  the  results  of  the  measurement  of  the  abilities  constructs  in  the  0*NET  data 
collection.  Finally,  we  discuss  the  implications  of  these  findings. 


Background 

Much  of  our  knowledge  of  human  abilities  derives  from  programmatic  factor  analysis 
research.  Critical  research  questions  in  the  realm  of  abilities  concern  the  generality  of  the 
constructs  used  to  describe  human  abilities.  Constructs  such  as  “mental  abilities,”  “motor 
abilities,”  “problem-solving  ability,”  “decision-making  ability,”  and  “agility”  have  turned  out  to 
be  too  broad;  the  tasks  required  by  such  broad  categories  are  too  diverse  to  yield  high 
correlations  between  performances  in  these  tasks.  However,  factor  analyses  of  the  correlations 


among  performances  within  these  domains  typically  yield  somewhat  more  narrowly  defined 
abilities  (see  e.g.,  Carroll,  1993;  Ekstrom,  French,  & Harman,  1976;  Guilford  & Hoepfner,  1971; 
Fleishman,  1964, 1972b).  Similarly,  expressions  like  “athletic  ability”  and  “musical  ability”  are 
often  used,  but  it  is  known  that  there  are  a number  of  separate  constructs  that  better  define 
several  different  abilities  involved  in  the  tasks  comprising  these  broad  activities.  On  the  other 
hand,  characterizing  an  individual  as  having  the  ability  to  “lift  barbells  of  a given  weight”  or  to 
“solve  quadratic  equations  of  a given  complexity”  yields  information  that  is  too  specific  and  not 

very  descriptive  of  an  ability  trait  that  extends  to  performance  in  a variety  of  tasks  requiring  the 
same  underlying  ability. 


Taxonomy 

Although  no  one  ability  taxonomic  system  can  meet  all  of  the  requirements  for  an 
occupational  information  system  which  has  the  scope  of  0*NET,  the  Ability  Requirements 
Taxonomy  developed  by  Fleishman  and  his  colleagues  provides  a foundation  that  meets  several 
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of  the  outlined  criteria.  The  taxonomy  has  a research  base  spanning  nearly  40  years  and  covers 
constructs  included  in  the  psychomotor,  physical,  cognitive,  and  sensory-perceptual  domains  of 
human  abilities.  The  job  analysis  measurement  system  based  on  this  taxonomy,  called  the 
Fleishman-Job  Analysis  Scales  (F-JAS;  Fleishman,  1975b,  1992)  now  has  a long  history  of  use 
and  evaluation  for  jobs  in  industry,  state  and  federal  government  agencies,  and  military 
occupational  specialties  (for  one  review  see  Fleishman,  1988).  The  system  has  been  used 
successfully  in  nationwide  job  analysis  studies  (e.g.,  Landy,  et  al.,  1992).  To  further  facilitate 
their  use  in  large  scale  administration,  these  scales  have  undergone  some  modifications  to  suit 
the  specific  purposes  of  the  present  effort. 

The  ultimate  objective  of  any  taxonomic  development  is  to  identify  the  most 
comprehensive,  but  parsimonious  categories  which  are,  at  the  same  time,  the  most  useful  and 
meaningful.  The  ability  taxonomy  developed  by  Fleishman  and  his  associates  (see,  e.g., 
Fleishman,  1964, 1972b;  Fleishman  & Quaintance,  1984;  Fleishman  & Reilly,  1992a,  1992b) 
followed  a programmatic  process  to  develop  a set  of  physical  and  psychomotor  abilities.  Briefly, 
a series  of  interlocking  experimental  and  factor  analytic  studies  involving  hundreds  of  tasks 
among  a variety  of  jobs  was  used  to  begin  development  of  the  taxonomy.  Experimental  studies 
designed  to  elicit  the  identification  and  definition  of  human  physical  and  psychomotor  abilities 
were  conducted  over  many  years  (see  Fleishman,  1954,  1956, 1958, 1964, 1966, 1967a,  1967b, 
1972b;  Fleishman  & Ellison,  1962;  Parker  & Fleishman,  1960;  Hempel  & Fleishman,  1955; 
Fleishman  & Reilly,  1992b;  Meyers,  Gebhardt,  Crump,  & Fleishman,  1993).  A project  sponsored 
by  the  U.S.  Defense  Advanced  Research  Projects  Agency  expanded  the  taxonomy  to  include 
cognitive  and  sensory-perceptual  categories  (Theologus  & Fleishman,  1973;  Theologus, 

Romas hko,  & Fleishman,  1973). 
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Subsequently,  this  initial  taxonomy  was  reviewed  and  refined  to  ensure  comprehensive 
coverage  of  all  ability  domains.  The  physical,  psychomotor,  cognitive,  and  sensory-perceptual 
abilities  were  combined  into  a single  list  and  operational  definitions  were  written  for  each  ability. 
This  provisional  list  was  reviewed  by  leading  psychologists  in  human  abilities  measurement  in  a 
series  of  discussions  and  interviews.  Feedback  from  the  reviewers  identified  areas  needing 
improvement  and  efforts  were  undertaken  to  clarify  the  definitions  and  include  more  task 
examples  for  each  category.  Further  research  and  literature  review  led  to  the  inclusion  of 
additional  categories  that  seemed  applicable  to  human  task  performance  such  as  Time  Sharing 
(Descriptor  #21)  and  Selective  Attention  (Descriptor  #20).  The  resulting  list  of  52  abilities 
comprised  the  taxonomy  that  was  incorporated  into  the  Manual  for  Ability  Requirements  Scales 
(MARS)  (Fleishman,  1975a,  1975b)  and,  later,  in  the  Fleishman  Job  Analysis  Survey  (F-JAS) 
(Fleishman,  1992). 

The  next  phase  of  the  programmatic  effort  was  to  develop  a measurement  system  for 
evaluating  the  ability  requirements  levels  of  various  jobs  and  job  tasks  using  this  taxonomy.  A 
series  of  expert  panels  and  interviews  (see  Fleishman,  1975b;  Fleishman  & Mumford,  1988, 

1991,  and  Fleishman  & Quaintance,  1984  for  a complete,  detailed  description)  resulted  in  the 
identification,  development,  and  refinement  of  behaviorally-anchored  rating  scales  for  each  of  the 
abilities  in  the  taxonomy.  High  and  low  behavioral  descriptions,  empirically  established  task 
examples,  and  the  ability  definitions  produced  in  the  initial  measurement  development  process 
were  presented  to  additional  expert  panels  for  comment  and  review.  The  final  product  of  this 
effort  was  the  F-JAS  ability  measurement  system,  consisting  of  52  abilities  spanning  the 
cognitive,  psychomotor,  physical,  and  sensory-perceptual  domains  of  human  performance 
capabilities. 
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As  noted  previously,  this  Ability  Requirements  Taxonomy  and  measurement  system  has 
been  extensively  evaluated.  While  a full  discussion  of  the  extensive  literature  supporting  the 
system  is  beyond  the  scope  of  this  chapter  (see  an  earlier  report  under  this  project  by  Fleishman, 
Wetrogan,  Uhlman,  & Marshall-Mies,  1995  and  a review  by  Fleishman  & Mumford,  1988; 
1991),  it  can  be  said  that  the  F-JAS  system  has  been  shown  to  have  substantial  reliability,  and 
internal  and  external  validity.  For  example,  the  F-JAS  typically  evidences  reliability  coefficients 
of  .80  and  above  in  a variety  of  occupations  when  15  or  more  judges  are  used  (e.g.,  Hogan, 
Ogden,  & Fleishman,  1978;  Myers,  Jennings,  & Fleishman,  1981).  Further,  these  reliabilities 
hold  up  across  rater  level  (incumbent,  supervisor,  subordinate)  and  generally  do  not  show 
common  rater  biases  such  as  the  halo  effect.  Additionally,  studies  show  high  agreement  between 
ability  profiles  provided  by  different  rater  groups  such  as  job  incumbents  and  occupational 
analysts. 

A variety  of  studies  have  demonstrated  aspects  of  the  validity  of  the  F-JAS.  For  example, 
Hogan,  Ogden  & Fleishman  (1979)  found  that  80  percent  of  the  tasks  performed  by  warehouse 
workers  could  be  assigned  to  one  or  more  of  the  ability  categories.  Similar  findings  were 
obtained  for  Army  officers  (Mumford,  Yarkin-Levin,  Korotkin,  Wallis,  & Marshall-Mies,  1985), 
FBI  special  agents  (Cooper,  Schemmer,  Jennings,  Korotkin,  1983)  and  for  New  York  City  Police 
officers  (Landy,  1988).  These  findings  provide  some  evidence  for  the  Ability  Requirements 
Taxonomy's  parsimony,  or  internal  validity.  The  external  validity  of  the  taxonomy  has  also  been 
amply  demonstrated.  For  example,  Hogan  et  al.  (1978)  found  that  ability  tests,  based  on  ability 
requirements,  derived  from  the  F-JAS  scales  produced  a multiple  R of  .45  in  predicting 
performance  on  the  job  sample.  Further,  Gebhardt  & Schemmer  (1985)  found  validities  in  the 
.80s  for  generic  tests  of  abilities  in  the  taxonomy  against  job  samples  of  tasks  performed  by  dock 
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workers.  More  recently,  the  ability  scales  have  been  used  to  define  the  ability  requirements  of 
managerial  jobs  (Friedman,  Fleishman,  & Fletcher,  1992)  and  to  predict  the  error  rates  associated 
with  tasks  performed  by  technicians  in  nuclear  power  plants  (Fleishman,  Buffardi,  Morath, 
McCarthy,  & Friedman,  1994).  Such  findings  are  but  one  approach  in  demonstrating  the  validity 
of  the  abilities  taxonomy  to  predict  task  performance,  a key  marker  of  external  validity. 

Thus,  the  selection  of  the  F-JAS  abilities  taxonomy  as  one  of  the  systems  for  gathering 
information  for  0*NET  is  well  supported,  empirically  and  rationally,  by  a variety  of  applications 
and  findings  from  research  efforts.  However,  in  order  to  ensure  the  applicability  and  utility  of  this 
taxonomy  for  the  present  effort,  the  entire  system  was  reassessed  in  light  of  the  demands  and 
requirements  of  the  project’s  objectives.  This  review  and  revision  process  in  discussed  below. 

Sample  and  Measures 

One  issue  arising  in  the  current  project  was  that  the  data  collection  effort  would  entail  not 
only  ability  ratings,  but  also  ratings  on  a variety  of  other  job  descriptors  (e.g.,  skills,  knowledges, 
work  activities)  across  a wide  range  of  occupations.  Accordingly,  there  would  likely  be  less  time 
available  for  completing  the  survey  and  it  was  possible  that  the  reading  levels  of  job  incumbents 
would  vary.  Thus,  several  modifications  were  made  to  adjust  reading  difficulty  and  instrument 
completion  time.  First,  some  ability  definitions  and  task  examples  were  revised  to  reduce  the 
reading  demands  on  the  raters.  Further,  a number  of  the  task  examples  used  to  anchor  the  rating 
scales  were  revised  to  include  more  occupationally-oriented  tasks  as  well  as  tasks  more  likely  to 
be  familiar  to  raters  regardless  of  their  job.  All  of  the  anchors  were  reviewed  for  appropriateness 
and  potential  cultural  bias,  as  well  as  to  ensure  that  they  could  be  understood  without  benefit  of 
specialized  experiences  or  knowledge.  It  is  worth  noting  that  new  or  revised  anchors  were 
empirically  rescaled  before  being  positioned  on  the  scales.  Finally,  the  original  F-JAS  tables 
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which  demonstrate  how  a particular  ability  is  different  from  other  abilities  with  which  it  might  be 
confused  were  removed  to  further  reduce  the  reading  time  and  reading  level  required.  The  revised 
scales  were  tested  during  an  initial  pilot  study  (Mumford  & Sager,  1995),  with  assistance  from 
the  Occupational  Analysis  Field  Center  (OAFC)  staff.  The  comments  of  OAFC  personnel  and  the 
data  received  from  this  administration  were  particularly  helpful  in  modifying  the  scales.  Figure  9- 
1 shows  the  names  of  the  52  ability  descriptors  organized  into  15  higher-order  aggregates  with 
four  highest-order  categories.  Figure  9-2  presents  a sample  page  from  the  questionnaire. 

In  the  data  collection  effort,  two  groups  of  judges  provided  ratings  on  the  abilities 
required  to  perform  the  target  occupations.  This  initial  data  collection  included  job  incumbents 
from  80  occupations.  For  the  analyses  reported,  only  occupations  for  which  at  least  four  (4) 
incumbents  provided  ratings  were  used.  This  criterion  resulted  in  a sample  of  613  incumbents 
from  33  occupations.  Figure  9-3  lists  these  occupations  and  the  numbers  of  incumbent 
respondents  completing  the  Abilities  Questionnaire  in  each  occupation. 

The  second  group  of  raters  included  occupational  analysts  provided  or  trained  by  the 

OAFCs.  In  this  sample,  a minimum  of  six  analysts  rated  the  ability  requirements  for  each  of  the 

80  target  occupations.  Because  incumbents’  ratings  for  only  33  occupations  were  available,  only 

the  analysts’  ratings  on  the  same  33  occupations  were  used  for  comparison  purposes.  The  analyst 

sample  totaled  347  raters,  although  many  analysts  rated  more  than  one  occupation  each. 

» 

The  incumbents  and  analysts  rated  each  of  the  abilities  on  two  scales.  First,  they  rated  the 
level  of  the  ability  required  for  the  occupation  on  a one  to  seven  scale.  This  scale  also  included  a 
“Does  not  apply”  option  to  allow  for  abilities  that  were  not  at  all  required  for  the  target 
occupation.  Second,  the  incumbents  and  analysts  rated  the  importance  of  the  ability  to  job 
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performance,  this  time  on  a one  to  five  scale.  If  a judge  rated  the  level  “zero,”  they  were 
instructed  to  skip  the  importance  rating. 

Results 

Descriptive  Statistics 

Table  9- 1 presents  the  means,  standard  deviations,  standard  error  of  measurements,  and 
interclass  correlations  obtained  for  the  level  and  importance  scales  for  each  of  the  52  abilities. 
The  means  are  averaged  across  the  33  occupations  for  which  there  were  at  least  four  respondents 
per  occupation.  As  these  are  averaged,  only  broad  interpretation  of  these  results  is  possible.  In 
terms  of  the  level  scale,  what  emerges  is  a general  trend  for  the  more  cognitive  abilities  to  be 
rated  higher  than  the  more  physical  ones.  For  example,  the  abilities  rated  highest  (on  the  seven 
point  level  scale)  across  occupations  include  Oral  Expression  (Descriptor  #3),  (M  = 4.58), 
Written  Comprehension  (Descriptor  #2)  (M  = 4.29),  and  Problem  Sensitivity  (Descriptor  #7)  (M 
= 4.21)  while  those  rated  lowest  include  Explosive  Strength  (Descriptor  #33)  (M  = 1.52),  Gross 
Body  Equilibrium  (Descriptor  #40)  (M  = 1.68),  and  Rate  Control  (Descriptor  #28)  (M  = 1.75). 
This  suggests  that  most  of  the  occupations  included  were  more  office-type  and/or  administrative, 
rather  than  physical,  outdoor  occupations.  The  standard  deviations  were  relatively  consistent 
across  all  the  abilities  with  most  being  at  or  around  2.0.  The  exceptions  here  were  the  more 
highly  rated  cognitive  abilities,  which  had  standard  deviations  of  approximately  1.75. 

Among  the  importance  ratings  (using  a five  point  scale),  a similar  pattern  emerges  with 
the  cognitive  abilities,  such  as  Oral  Comprehension  (Descriptor  #1)  (M  = 3.70)  and  Written 
Expression  (Descriptor  #4)  (M  = 3.40)  rated  higher  than  most  of  the  physical  abilities,  including 
Dynamic  strength  (Descriptor  #34)  (M  = 1.81)  and  Dynamic  Flexibility  (Descriptor  #38)  (M  = 
1.90).  Again,  the  standard  deviations  were  generally  around  1.20,  with  the  cognitive  abilities  a 
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little  lower  (around  1.1)  and  the  physical  abilities  a little  higher  (approximately  1.2  to  1.3).  These 
findings  also  support  the  conclusion  that  the  occupational  sample  represents  predominately 
indoor,  office-type  occupations. 

Reliability 

Table  9-1  also  presents  the  intraclass  correlations  (reliabilities)  of  the  abilities.  Overall, 
these  reliabilities  reflect  general  agreement  among  the  job  incumbents  across  occupations.  Most 
of  the  reliabilities  for  both  the  level  and  importance  scales  were  sufficiently  high,  generally  above 
.80  (range  .33  to  .91;  for  level,  M = .79;  for  importance,  M = .75).  These  high  overall  reliabilities 
led  to  relatively  low  standard  error  of  measurement  results,  with  almost  all  being  below  1.0  for 
level  and  .75  for  importance.  From  these  findings,  we  can  conclude  that  the  abilities  showed 
acceptably  high  reliabilities  across  occupations  and  that  there  is  some  variation  between  the 
occupations  as  to  the  level  and  importance  of  these  human  abilities. 

In  order  to  assess  both  the  lower  limits  of  reliability  and  optimal  reliabilities,  the 
reliability  for  a single  rater  was  estimated  as  was  the  reliability  for  30  raters.  Table  9-2  shows  the 
reliability  for  each  ability  transformed  into  single  and  30-rater  estimates.  The  single  rater 
reliability  estimates  are,  for  the  most  part,  low  to  moderate,  with  level  reliabilities  ranging  from 
.06  to  .46  and  importance  from  .04  to  .49.  The  Spearman-Brown  corrected  reliabilities  for  30 
raters  are,  as  might  be  expected,  much  larger,  with  almost  all  coefficients  for  both  level  and 
importance  in  the  .80s  and  .90s. 

Scoring 

An  analysis  was  run  in  an  attempt  to  determine  whether  the  number  of  points  on  the 
rating  scales  had  any  impact  on  the  ratings.  In  order  to  investigate  this  possibility,  reliability 
estimates  were  calculated  on  the  incumbent  data  for  three  variations  of  the  level  scale:  zero  (not 
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relevant)  to  seven  (high),  one  (low)  to  seven  (high)  (i.e.,  not  relevant  responses  omitted),  and 
zero  (not  relevant)-one  (relevant).  In  the  first  two  cases,  the  reliabilities  were  sufficiently  high 
and,  in  fact,  did  not  differ  substantially.  The  average  difference  between  ^ and  & for  both  level 
and  importance  was  about  .01.  The  binary  coding  scheme  for  level,  rc,  was  about  .12  lower,  on 
average,  than  the  full  scale  reliability  coefficient,  suggesting  that  it  does  appear  to  have 
significantly  lower  reliability. 

Analyses  of  Variance 

Another  way  of  looking  at  the  reliability  of  the  ratings  is  to  assess  the  amount  of  variance 
that  the  abilities  and  the  occupations  being  rated  account  for  in  an  ANOVA  framework. 
Accordingly,  ANOVAs  were  run  for  the  abilities  across  occupations  and  these  results  are 
presented  in  Tables  9-4a  and  9-4b.  These  findings  showed  that  the  ability  ratings  did  significantly 
differentiate  the  occupations  for  both  level  (F  = 5.84,  p < .05)  and  importance  (F  = 4.99,  p < .05). 
Furthermore,  there  were  both  significant  differences  between  abilities  (as  might  be  expected)  as 
well  as  interactions  between  occupations  and  abilities  for  both  scales.  These  findings  replicated 
when  the  abilities  were  grouped  into  their  higher-order  taxonomy  (see  Tables  9-6a  and  9-6b). 

That  is,  mean  ratings  were  calculated  by  combining  scales  according  to  15  higher-order  groups 
(e.g.,  verbal  abilities,  which  included  four  scales  of  oral  and  written  comprehension  and 
expression). 

Table  9-5  shows  the  reliabilities  as  calculated  using  the  results  of  Tables  9-4a  and  9-4b, 
with  the  descriptor  by  occupations  effect  as  true  variance,  and  descriptor  by  occupations  within 
subjects  as  error  variance.  These  estimates  were  very  high  across  the  board  with  the  actual  rater 
results  for  level  and  importance  (r*  = .82  for  both)  being  substantial.  Again,  Spearman-Brown 
was  used  to  calculate  reliability  estimates  for  one  and  30  raters.  The  findings  here  echo  the 
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above,  as  the  one  rater  estimates  for  level  and  importance  are  .32,  and  the  30  rater  estimates  are 
both  .93.  These  results  lead  to  the  conclusion  that,  if  there  are  a sufficient  number  of  incumbents 
rating  a given  occupation  for  the  abilities,  they  can  be  reliably  rated  by  individuals  in  a variety  of 
occupations  and  the  mean  rating  profiles  differentiate  occupations.  Table  9-7  shows  similar 
effects  for  aggregated  scales. 

What  can  be  concluded  from  these  analyses  is  that  the  abilities,  individually, 
incrementally  aggregated,  and  as  a whole,  were  able  to  reliably  account  for  a significant  amount 
of  the  variance  both  within  and  across  occupations.  Additionally,  these  results  can  be  considered 
as  further  support  for  the  previously  established  reliability  and  validity  of  the  ability  requirements 
scales  (Fleishman,  1992). 

Descriptor  and  Scale  Relationships 

Table  9-8  shows  the  correlations  between  ratings  made  on  the  level  and  importance 
scales.  The  two  scales  correlated  .82,  on  average,  when  the  correlations  are  calculated  within 
each  occupation  (across  the  52  ability  scales)  and  show  considerable  variation  in  their 
relationship  across  the  33  occupations  (SD  = .30).  The  average  correlation  was  ten  points  higher 
(.92)  when  calculated  within  each  descriptor  (across  the  33  occupations)  and  showed  much  less 
variation  across  the  52  descriptors  (SD  = .07).  We  interpret  this  to  mean  that,  for  a considerable 
number  of  occupations,  the  importance  and  level  scales  are  providing  non-redundant 
information-some  abilities  may  be  at  relatively  low  levels,  but  still  be  important  for  a given 
occupation.  On  the  other  hand,  level  and  importance  ratings  do  provide  highly  similar 
information  for  a given  descriptor  when  looked  at  across  occupations. 

Tables  9-9a,  9-9b,  9-10a,  and  9-l0b  show  the  correlations  among  the  abilities  for  the 
level  and  importance  scales,  for  occupation-level  (correlating  mean  occupation  ratings)  and 
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individual-level  ratings  (made  by  four  individuals  randomly  selected  for  each  of  the  33 
occupations).  These  52  by  52  matrices  provide  a wealth  of  information,  only  a small  part  of 
which  can  be  discussed  herein.  However,  the  primary  conclusion  from  these  correlational 
analyses  is  that  they  further  support  the  meaningfulness  and  coherence  of  the  ability  scales  as 
occupation  descriptors.  For  example,  for  the  level  ratings  of  ability  levels  required,  we  again  see 
that  abilities  tend  to  “cluster”  together  as  expected,  with  groups  of  highly  correlated  cognitive 
abilities  (e.g..  Oral  Expression  [Descriptor  #3]  and  Written  Expression  [Descriptor  #4],  r = .80), 
psychomotor  (e.g..  Perceptual  Speed  [Descriptor  #17]  and  Selective  Attention  [Descriptor  #20], 
= .68)  and  physical  (e.g..  Static  Strength  [Descriptor  #32]  and  Dynamic  strength  (Descriptor 
#34),  r = .93)  all  in  evidence.  The  correlations  among  importance  ratings  also  reflect  similar 
patterns.  While  other  examples  could  be  cited,  these  results,  in  general,  provide  evidence  for  the 
meaningfulness  of  the  abilities  as  they  relate  to  each  other  and  as  they  work  together  to  describe 
occupational  requirements. 

Factor  Structure 

T o summarize  these  between-scale  relationships,  we  conducted  a factor  analysis  on  the 
ability  level  ratings.  The  ability  ratings  were  entered  into  a principal  components  analysis.  The 
results,  shown  in  Table  9-11,  suggested  that  seven  components/factors  should  be  retained.  The 
first  factor  was  a broad  psychomotor/perceptual  factor  and  included  a number  of  the  physical 
abilities  as  well.  Several  of  the  abilities  loaded  above  .90  on  this  factor  including  Depth 
Perception  (Descriptor  #46)  (r  = .95)  and  Rate  Control  (Descriptor  #28)  (r  = .92),  along  with 
Dynamic  strength  (Descriptor  #34)  (r  = .88)  and  Explosive  Strength  (Descriptor  #33)  (r  = .85). 
Given  that  there  were  27  abilities  loading  above  .60  on  this  factor,  it  is  clearly  a kind  of  “g” 
ability  factor  covering  the  basic  physical  and  psychomotor  areas.  Factor  number  two  was  a 
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relatively  broad  cognitive  factor,  with  high  loadings  on  Deductive  Reasoning  (Descriptor  #8)  (r  = 
.95),  Written  Expression  (Descriptor  #4)  (r  = .93),  and  Written  Comprehension  (Descriptor  #2)  (r 
= .89),  along  with  the  other  cognitive  abilities.  Once  again,  this  general  factor  had  a large  number 
(18)  of  abilities  loading  above  .50.  The  third  factor  was  marked  by  high  Visualization  (Descriptor 
#19)  (r  = .71),  Finger  Dexterity  (Descriptor  #24)  (r  = .60),  and  Perceptual  Speed  (Descriptor  #17) 
(r  = .59)  and  was  termed  visualization.  Given  the  nature  of  the  occupations  sampled,  this  group 
probably  represents  a cluster  of  particular  occupations  in  the  study.  Factor  four,  time  sharing,  was 
characterized  primarily  by  the  loading  on  Time  Sharing  (Descriptor  #21)  (r  = .57),  along  with 
Memorization  (Descriptor  #14)  (r  = .51)  and  Selective  Attention  (Descriptor  #20)  (r  = .52).  Since 
the  two  abilities  marking  factor  five  were  Speech  Recognition  (Descriptor  #51)  (r  = .86)  and 
Speech  Clarity  (Descriptor  #52)  (M  = .72),  this  factor  was  clearly  a speech  factor.  The  last 
interpretable  factor  was  termed  wrist-finger  speed,  from  the  loadings  of  abilities  such  as  Wrist- 
Finger  Speed  (Descriptor  #30)  (r  = .84),  as  well  as  Near  Vision  (Descriptor  #41)  (r  = .59)  and 
Perceptual  Speed  (Descriptor  #17)  (r  = .47).  The  last  factor  appeared  to  be  a non-interpretable, 
residual  factor.  Overall,  these  seven  factors  were  able  to  account  for  59.72%  of  the  variance. 

Occupation  Differences 

To  illustrate  the  degree  to  which  abilities  distinguish  occupations,  we  looked  at  the 
descriptive  statistics  (means  and  standard  deviations)  for  a sample  of  specific  occupations  for 
which  the  abilities  might  prove  useful.  As  with  other  domains  of  0*NET  descriptors,  the 
representative  occupations  included  General  Managers  and  Top  Executives,  Computer 
Programmers,  Registered  Nurses,  Police  Patrol  Officers,  Janitors  and  Cleaners,  and  Maintenance 
Repairers,  General  Utility.  The  results  for  the  level  and  importance  scales  appear  in  9-12a  and  9- 
12b.  From  these  results,  several  broad  trends  emerge.  First,  as  noted  above,  the  more  cognitive 
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abilities  were  rated  higher  on  level  for  most  of  the  representative  occupations.  For  example. 
Written  Comprehension  (Descriptor  #2)  was  rated  high  by  Managers  (M  = 5.21),  Police  Officers 
(M  = 4.70),  and  Maintenance  Workers  (M  = 4.26)  alike.  The  only  occupation  not  generally  rated 
above  the  midpoint  on  this  and  other  cognitive  abilities  (e.g..  Oral  Expression  [Descriptor  #3], 
Problem  Sensitivity  [Descriptor  #7])  was  Janitors/Cleaners.  On  the  other  hand,  the  physical  and 
psychomotor  abilities  were  generally  rated  low  by  several  of  these  occupations,  with  Computer 
Programmers’  ratings  of  Gross-Body  Equilibrium  (Descriptor  #40)  (M  = .50)  and  Nurses’  ratings 
of  Rate  Control  (Descriptor  #28)  (M  = 1.43)  almost  negligible.  As  expected  in  the  more  physical 
occupations,  such  as  Police  Officers  and  Maintenance  Repairers,  there  were,  of  course,  higher 
ratings  on  these  abilities.  Among  them.  Police  Officers  rated  abilities  like  Reaction  Time 
(Descriptor  #29)  (M  = 6.00),  Response  Orientation  (Descriptor  #27)  (M  = 5.00),  and  Night 
Vision  (Descriptor  #44)  (M  = 5.09)  very  high,  as  did  Maintenance  Repairers’  rate  on  Static 
Strength  (Descriptor  #32)  (M  = 4.87)  and  Control  Precision  (Descriptor  #25)  (M  = 4.26). 

The  second  interesting  finding  is  that  the  pattern  of  responses  within  occupations  seemed 
to  do  a very  good  job  of  describing  those  occupations.  For  example.  Computer  Programmers 
rated  the  more  cognitive  abilities  at  the  mid-point  of  the  level  scale  with  most  means  in  the  upper 
fours  and  fives.  Conversely,  almost  all  of  the  concrete,  physical  abilities  were  rated  near  or  below 
one,  reflecting  the  sedentary  nature  of  their  work.  Police  Officers,  on  the  other  hand,  displayed  a 
more  stable,  even  distribution  across  the  abilities,  with  many  of  the  cognitive,  psychomotor,  and 
physical  ability  groups  evidencing  level  ratin'*  above  four.  The  only  abilities  rated  exceptionally 
low  by  Police  Officers  were  several  narrow  tfive  abilities  such  as  Information  Ordering 
(Descriptor  #1 1)  (M  = .87)  and  Math  Reasoni,^  (Descriptor  #12)  (M  = 1.65).  Janitors  showed  a 
similar  pattern  of  response  as  the  Police  Officers,  but  at  a lower  mean  level.  There  responses 
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were  relatively  similar  across  all  the  abilities,  but  in  only  a few  cases  did  any  of  the  level  ratings 
exceed  three  and  none  exceeded  three  and  a half. 

The  results  for  the  importance  ratings  oh  the  abilities  for  these  six  occupations  appear  in 
Table  9- 12b,  and  these  are  generally  similar  to  those  for  the  level  scale.  The  level  and  importance 
scales  were  correlated  at  .92  across  descriptors  within  occupations  and  .82  across  occupations 
within  descriptors  (see  Table  9-8),  so  this  is  not  too  surprising.  Once  again.  Managers  and 
Programmers  rated  the  cognitive  abilities  as  more  important  and  the  physical  ones  less  so,  while 
Police  Officers  rated  most  of  them  across  the  board  as  at  least  somewhat  important.  Nurses, 
Police  Officers,  Janitors,  and  Maintenance  Repairers  evidenced  similar  response  patterns  as  for 
the  level  scale  results. 

Discriminant  Analysis 

A discriminant  function  analysis  was  run  to  identify  the  way  in  which  the  level  ratings  on 
abilities  differentiated  occupations.  For  this  analysis,  the  level  ratings  for  each  of  the  abilities 
were  used  as  independent  variables  and  occupational  membership  was  used  as  the  dependent 
variable.  Each  incumbent  was  reclassified  into  one  occupation,  using  the  discriminant  functions 
identified  in  the  analysis.  The  analysis  found  five  significant  functions.  Using  all  20  functions, 
60%  of  the  total  variance  was  accounted  for,  and  62%  of  incumbents  correctly  classified  into 
their  occupations,  suggesting  some  utility  in  discriminating  occupations.  Table  9-13  shows  the 
five  significant  functions,  the  sum  of  squared  rotated  correlations  of  the  descriptor  ratings  with 
the  functions,  and  the  T|"  values  for  the  descriptor  ratings.  The  five  functions  were  quite 
interpretable  based  on  the  ability  loadings.  The  first  function,  physical,  was  characterized  by 
abilities  such  as  Static  Strength  (Descriptor  #32)  (r  = .73)  and  Gross-Body  Coordination  (r  = .60), 
while  the  second  function,  cognitive,  had  high  loadings  on  abilities  such  as  Written  Expression 
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(Descriptor  #4)  (r  = .62)  and  Fluency  of  Ideas  (Descriptor  #5)  (r  = .54).  The  third  function  was 
unique  in  that  it  had  only  one  significant  ability  loading  on  it,  Wrist-Finger  Speed  (Descriptor 
#30)  (r  = .69),  so  is  labeled  wrist-finger  speed.  The  fourth  function  was  a psvchomotor-sneeH  • 
group,  characterized  by  Reaction  Time  (Descriptor  #29)  (r  = .50)  and  Rate  Control  (Descriptor 
#28)  (r  = .49)  and  the  final  significant  function  was  termed  mathematics  due  to  the  loadings  of 
Number  Facility  (Descriptor  #13)  (r  = .65)  and  Math  Reasoning  (Descriptor  #12)  (r  = .59).  The 
sum  of  the  squared  function  coefficients  are  mostly  in  the  .30s  to  .40s  and  the  i\2  coefficients 
range  from  .09  to  .37.  The  coefficients  for  both  of  these  indices  demonstrate  the  overall 
contribution  of  many  of  the  individual  abilities  to  discriminating  occupations. 

Convergence  With  Analysts’  Ratings 

The  fact  that  we  were  able  to  collect  ratings  from  both  job  incumbents  and  occupational 
analysts  allowed  for  some  comparisons  between  these  two  groups’  views  of  the  target 
occupations.  The  results  of  these  comparisons  for  both  the  level  and  importance  scales,  including 
means,  standard  deviations,  N and  F tests,  along  with  correlations  and  d2,  are  shown  in  Tables  9- 
14a  and  9- 14b.  Overall,  both  the  analysts  and  the  incumbents  were  generally  in  agreement  with 
regards  to  their  ratings  assigned  to  the  abilities.  The  correlations  ranged  from  .37  to  .88  for  level, 
with  the  mean  of  .70  and  majority  of  correlations  in  the  .70s  and  .80s.  For  importance,  the 
correlations  ranged  from  .14  to  .88,  again,  mostly  in  the  .70s  and  .80s,  and  the  mean  was  .65. 

The  occupational  analysts’  level  ratings  were  more  reliable  than  those  of  the  incumbents 
(Manalysts  = *87  vs.  Mincum.  = .79),  even  considering  that  there  were  fewer  analysts  rating  these 
occupations  (Manalysts = 10.51  vs.  Mincum. = 18.57).  The  importance  scale  showed  a similar  pattern, 
with  analyst  reliabilities  slightly  higher  than  those  for  incumbents. 
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There  were  some  significant  differences  between  analysts  and  incumbents  in  the  actual 
level  ratings  of  certain  individual  abilities,  as  evidenced  by  the  significant  t-test  results  and 
significant  F tests  for  variance  differences.  Practically  speaking,  though,  these  differences  were 
not  large.  The  mean  d was  about  one  point  for  level  and  .40  for  importance,  indicating  that  the 
differences  between  analysts  and  incumbents  were  not  large,  on  average.  A few  abilities  did 
show  larger  differences,  notably  Selective  Attention  (Descriptor  #20),  Time  Sharing  (Descriptor 
#21),  and  Descriptors  #49  through  #52,  having  to  do  with  auditory  and  speech  recognition 
abilities. 

What  these  differences  show  is  that  for  level,  the  incumbents  in  the  samples  rated  more 
abilities  higher  than  did  the  analysts.  Interestingly,  this  pattern  reversed  itself  for  the  importance 
ratings,  with  the  analysts  rating  the  abilities  as  more  important  than  did  the  incumbents. 

However,  the  major  finding  was  the  high  correlations  between  the  mean  ratings  of  analysts  and 
those  received  from  incumbents  across  different  jobs. 

Since  the  ratings  obtained  from  incumbents  and  occupational  analysts  were  generally  in 
agreement  (although  there  were  some  mean  level  rating  differences),  we  ran  a principal 
components  analysis  on  the  analyst  data  to  see  how  it  compared  to  the  incumbent  factor  structure. 
This  analysis  was  run  with  the  same  assumptions  and  rotation  procedures  carried  out  on  data 
from  the  incumbents.  The  factor  analysis  data  from  the  analysts  are  presented  in  Table  9-16. 
While  there  was  a great  deal  of  overlap  between  the  two  solutions,  there  were  some  disparities 
that  are  worth  noting.  Two  factors  were,  for  the  most  part,  the  same  for  the  two  solutions.  For 
example,  factor  one  for  the  analysts  was  a broad  cognitive  component  with  high  loadings  on 
Deductive  Reasoning  (Descriptor  #8)  (r  = .94)  and  Oral  Comprehension  (Descriptor  #1)  (r  = .93) 
among  the  highest.  The  second  factor  was  a broad  psvchomotor/perceptual  one  characterized  by 
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Response  Orientation  (Descriptor  #27)  (r  = .92)  and  Rate  Control  (Descriptor  #28)  (r  = .90).  The 
first  two  incumbent  factors  were  a psychomotor/perceptual  and  then  a cognitive  factor.  Analyst 
factor  three  was  a clear  physical  ability  factor,  marked  by  Extent  Flexibility  (r  = .83)  and  Trunk 
Strength  (Descriptor  #35)  (r  = .79).  Several  other  abilities,  with  loadings  on  the 
perceptual/psychomotor  factor  in  the  factor  analysis  of  incumbent  ratings,  also  loaded  highly  on 
this  analyst  factor  (e.g.,  Explosive  Strength  [Descriptor  #33],  .65,  and  Gross  Body  Coordination 
[Descriptor  #39],  .59). 

The  fourth  analysts’  factor  was  called  sensorv/dexteritv  as  it  was  marked  by  Visual  Color 
Discrimination  (Descriptor  #43)  (r  = .75),  Finger  Dexterity  (Descriptor  #24)  (r  = .63),  and  Arm- 
hand  Steadiness  (Descriptor  #22)  (r  = .61).  This  factor  had  no  comparable  counterpart  in  the 
analysis  based  on  incumbents.  Rather,  the  abilities  in  this  factors  loaded  in  a broader 
psychomotor  factor.  Factor  five  was  also  similar  to  the  incumbents’  results,  as  it  was  marked  by 
Wrist-Finger  Speed  (Descriptor  #30)  (r  = .80)  and  Flexibility  of  Closure  (Descriptor  #16)  (r  = 
.74)  and  is  called  wrist-finger  speed.  Finally,  the  sixth  factor,  speech,  mirrors  another  incumbent 
factor,  that  of  speech  recognition. 

Based  on  the  above  results,  the  incumbents  seemed  to  provide  slightly  less  reliable  and 
somewhat  higher  level  ratings.  On  the  other  hand,  the  reliabilities  are  still  high  and  the  pattern  of 
relationships  and  the  tightness  of  the  factor  structure  seemed  clearer  for  the  incumbents  than  for 
the  analysts.  It  is  interesting  to  note  that  this  pattern  is  similar  to  that  obtained  in  comparing  the 
incumbents  and  analysts  on  the  knowledge  ratings  (see  Chapter  4).  Although  these  findings  are 
qualitative  interpretations,  they  do  provide  some  additional  evidence  for  the  general  similarity  of 
the  incumbents  and  analysts’  ratings.  However,  they  also  raise  the  possibility  that  the  analysts 


Abilities  9-19 


and  the  raters  have  somewhat  different  views  of  at  least  some  of  the  ability  requirements  of  these 
target  occupations. 

To  see  if  analyst  and  incumbent  ratings  differed  depending  on  the  scale  coding,  means, 
standard  deviations,  reliabilities,  and  t-tests  were  calculated  for  each  ability  using  the  relevant/not 
relevant  (zero-one)  coding  scheme.  The  results  of  this  analysis  are  presented  in  Table  9-15.  For  a 
number  of  descriptors  (e.g..  Oral  Comprehension  [Descriptor  #1]  and  Number  Facility 
[Descriptor  #13]),  analysts  were  unanimous  in  their  judgment  that  the  abilities  were  relevant  for 
all  occupations,  while  incumbents  varied  in  their  ratings.  Across  occupations,  the  analysts  were 
more  likely  than  incumbents  to  judge  abilities  to  be  relevant.  However,  examination  of  the 
reliabilities  reveal  that  incumbents’  ratings  generally  are  at  least  as  reliable  as  the  analysts’ 
ratings.  In  fact,  they  are  often  more  reliable  than  the  analysts’  ratings  (e.g..  Information  Ordering 
[Descriptor  #10],  Speed  of  Closure  [Descriptor  #15],  and  Visualization  [Descriptor  #19]).  These 
results  are  consistent  with  the  hypothesis  that  incumbents’  ratings  show  more  differentiation 
among  occupations  than  analysts’  ratings,  based  oh  the  relevance  of  abilities. 

Conclusions 

This  chapter  has  summarized  the  development  of  the  ability  taxonomy  and  associated 
measurement  system  for  determining  the  ability  requirements  of  jobs.  The  taxonomy,  which 
spans  the  cognitive,  psychomotor,  physical,  and  sensory-perceptual  domains  of  human 
performance  capabilities,  is  based  on  earlier  work  of  Fleishman  and  associates  and  adapted  to  the 
requirements  of  the  0*NET  system.  For  the  present  effort,  special  attention  was  given  to  making 
the  measures  more  readable,  understandable,  and  user-friendly  so  that  they  would  be  as 
applicable  as  possible  across  the  wide  range  of  target  occupations. 
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The  results  of  this  study  confirm  the  findings  from  earlier  work  with  this  taxonomy  and 
measurement  system.  They  indicate  that  the  ability  requirements  approach  provides  reliable, 
useful  descriptive  and  interpretative  information  that  contribute  to  the  understanding  and 
measurement  of  the  kinds  and  levels  of  human  abilities  required  in  a wide  range  of  occupations. 
The  fact  that  the  profiles  of  ability  ratings  were  interpretable  for  both  top  Managers  and  Janitors 
is  but  one  piece  of  evidence  supporting  this  conclusion.  Although  only  33  occupations  were 
represented  in  these  analyses,  the  consistency  and  coherence  across  the  occupations  and  types  of 
ratings  makes  it  reasonable  to  conclude  that  the  abilities  taxonomy  would  prove  useful  for  other 
occupations  not  included  herein. 

The  data  collection  and  rating  methodology  for  determining  both  the  level  and  importance 
of  these  abilities  in  performing  these  occupations  was  shown  to  have  high  reliabilities  and 
internal  validity.  Missing  from  the  present  effort  is  additional  evidence  for  the  criterion-related 
validity  of  the  ratings.  Certainly,  one  avenue  for  future  research  would  be  to  gather  job 
performance  criteria  and  demonstrate  the  predictive  power  derived  from  the  ability  ratings  as 
well  as  from  the  other  descriptive  systems  developed  in  this  project.  However,  we  do  know  that 
earlier  studies  have  shown  that  tests  and  assessments  developed  on  the  basis  of  ability 
requirements  derived  from  this  system  have  high  criterion  related  validity  (e.g.,  Hogan,  1978; 
Gebhardt  & Schemmer,  1985).  Thus,  we  can  be  reasonably  confident  that  additional  criterion 

studies  will  further  support  the  body  of  research  demonstrating  the  overall  validity  of  the  abilities 
taxonomy. 

With  respect  to  the  reliability  of  the  ratings,  it  was  shown  that  job  incumbents  and 
occupational  analysts  showed  a high  degree  of  agreement  on  the  profiles  of  abilities  derived  to 
describe  the  ability  requirements  of  different  occupations.  While  there  were  some  differences  in 
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mean  scores  in  particular  abilities,  the  reliabilities  of  ratings  of  individual  ability  requirements 
were  mostly  high.  The  discriminant  function  analyses  and  factor  analyses  supported  the  internal 
validity  of  the  system,  but  interpretations  are  necessarily  constrained  by  the  particular  mix  of 
occupations  in  the  sample.  However,  there  was  ample  evidence  that  the  system  differentiates 

i 

occupations  in  a meaningful  way. 

With  respect  to  the  rating  scale  formats  evaluated,  there  were  relatively  high  correlations 
between  level  and  importance  scales  and  comparable  reliabilities.  Further,  regardless  of  the 
scoring  scheme  used,  these  reliabilities  remained  reasonably  high  (except  for  the  binary-coded 
scale),  and  there  was  little  difference  between  analysts  and  incumbents  across  coding  approaches. 
Given  the  relatively  high  correlations  across  occupations  and  individuals  between  the  level  and 
importance  and  level  ratings,  the  utility  of  using  both  scales  in  data  collection  is  a question  that  is 
worthy  of  further  attention.  On  balance,  however,  it  appears  that  the  “level”  scale  has  more 
advantages  conceptually  for  future  use. 

In  summary,  it  appears  that  the  content,  structure,  and  scaling  methodology  developed 
should  be  useful  in  areas  such  as  job  analysis,  person-job  matching,  occupational  and  career 
counseling,  and  the  development  of  job  families  having  similar  ability  requirements.  It  should  be 
noted  that  the  present  effort  was  somewhat  limited  in  the  number  of  occupations  and  incumbents 
represented.  However,  the  diversity  of  the  included  occupations  along  with  the  opportunity  to 
compare  the  ratings  of  incumbents  and  analysts  provide  some  assurance  that  the  above  results  are 
both  representative  and  generalizable  to  the  larger  population  of  occupations.  That  said,  we  feel 
safe  in  concluding  that  the  ability  taxonomy  and  scaling  methodology  should  assist  in  the  overall 
effort  to  develop  a more  complete  understanding  of  the  requirements  of  human  work. 
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Table  9-1 

Descriptive  Statistics  Across  All  Occupations  and  Reliability  Estimates  for  Rated  Differences 
Between  Occupations:  Abilities 

Variable 


Level Importance 


Descriptor 

M 

SD 

SEMa 

Lcb 

M 

SD 

SEM 

A 

1.  Oral  Comprehension 

4.31 

1.66 

.82 

.75 

3.70 

1.06 

.61 

.66 

2.  Written  Comprehension  4.29 

1.73 

.70 

.83 

3.59 

1.06 

.52 

.75 

3.  Oral  Expression 

4.58 

1.69 

.83 

.73 

3.71 

1.06 

.56 

.71 

4.  Written  Expression 

3.98 

1.85 

.65 

.87 

3.40 

1.11 

.42 

.85 

5.  Fluency  of  Ideas 

2.77 

1.97 

.79 

.84 

2.51 

1.18 

.51 

.81 

6.  Originality 

3.34 

1.99 

.82 

.83 

2.72 

1.18 

.52 

.81 

7.  Problem  Sensitivity 

4.21 

1.86 

.71 

.85 

3.41 

1.16 

.50 

.81 

8.  Deductive  Reasoning 

3.64 

1.96 

.80 

.83 

3.01 

1.17 

.55 

.78 

9.  Inductive  Reasoning 

3.37 

2.08 

.73 

.88 

2.79 

1.23 

.44. 

.87 

10.  Information  Ordering 

3.75 

1.87 

1.00 

.70 

3.19 

1.17 

.82 

.50 

1 1.  Category  Flexibility 

2.46 

2.17 

1.10 

.74 

2.28 

1.23 

.69 

.68 

12.  Math  Reasoning 

2.74 

2.01 

.86 

.82 

2.57 

1.23 

.58 

.78 

13.  Number  Facility 

3.61 

1.86 

.79 

.82 

3.08 

1.19 

.60 

.74 

14.  Memorization 

4.06 

1.66 

1.15 

.52 

3.25 

1.05 

.85 

.33 

15.  Speed  of  Closure 

3.27 

1.92 

.91 

.77 

2.71 

1.15 

.60 

.73 

16.  Flexibility  of  Closure 

2.45 

2.22 

1.22 

.69 

2.25 

1.25 

.78 

.61 

17.  Perceptual  Speed 

2.76 

2.08 

1.54 

.45 

2.46 

1.22 

.99 

.34 

18.  Spatial  Orientation 

2.68 

2.25 

1.27 

.68 

2.36 

1.27 

.68 

.71 

19.  Visualization 

2.79 

2.11 

1.16 

.69 

2.42 

1.22 

.79 

.58 

20.  Selective  Attention 

3.98 

1.88 

1.51 

.63 

3.23 

1.13 

.69 

.62 

21.  Time  Sharing 

4.10 

1.95 

1.13 

.66 

3.16 

1.17 

.77 

.57 

22.  Arm-hand  Steadiness 

2.26 

2.32 

.94 

.83 

2.12 

1.28 

.47 

.86 

23.  Manual  Dexterity 

2.63 

2.29 

.98 

.82 

2.31 

1.33 

.54 

.83 

24.  Finger  Dexterity 

2.41 

2.20 

1.27 

.66 

2.21 

1.22 

.70 

.67 

25.  Control  Precision 

2.13 

2.18 

.81 

.86 

2.07 

1.21 

.45 

.86 

26.  Multi-limb 

2.14 

2.25 

.87 

.85 

2.05 

1.23 

.45 

.86 

Coordination 

27.  Response  Orientation 

2.50 

2.28 

.85 

.86 

2.29 

1.32 

.49 

.86 

28.  Rate  Control 

1.75 

2.26 

.69 

.90 

1.83 

1.19 

.34 

.92 

29.  Reaction  Time 

2.63 

2.63 

.92 

.88 

2.37 

1.48 

.57 

.87 

30.  Wrist  -finger  Speed 

3.50 

2.25 

1.01 

.80 

2.78 

1.34 

.55 

.83 

31.  Speed  of  Limb 

2.45 

2.43 

.95 

.84 

2.17 

1.31 

.50 

.85 

Movement 

32.  Static  Strength 

2.38 

2.32 

.70 

.91 

2.14 

1.25 

.37 

.91 

33.  Explosive  Strength 

1.52 

2.15 

.69 

.90 

1.70 

1.13 

.37 

.89 

34.  Dynamic  Strength 

1.61 

2.13 

.76 

.87 

1.81 

1.16 

.44 

.85 

35.  Trunk  Strength 

2.14 

2.05 

.87 

.82 

2.15 

1.20 

.57 

.77 

36.  Stamina 

1.90 

2.03 

.79 

.85 

2.01 

1.19 

.47 

.85 
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Table  9-1  (continued) 

Descriptive  Statistics  Across  All  Occupations  and  Reliability  Estimates  for  Rated  Differences 
Between  Occupations:  Abilities 

Variable 


Level 

Importance 

Descriptor 

M 

SD 

SEM 

£k 

M 

SD 

SEM 

hr 

37.  Extent  of  Flexibility 

2.52 

2.10 

.81 

.85 

2.25 

1.20 

.47 

.85 

38.  Dynamic  Flexibility 

1.80 

2.01 

.83 

.83 

1.90 

1.13 

.48 

.82 

39.  Gross  Body 

2.01 

2.03 

.77 

.86 

2.04 

1.17 

.47 

.84 

Coordination 

40.  Gross  Body 

1.68 

2.09 

.75 

.87 

1.89 

1.19 

.43 

.87 

Equilibrium 

41.  Near  Vision 

3.63 

1.92 

1.15 

.66 

2.90 

1.16 

.80 

.51 

42.  Far  Vision 

2.52 

2.16 

.91 

.82 

2.26 

1.19 

.52 

.81 

43.  Visual  Color  Discrimin 

2.21 

2.20 

.88 

.84 

2.16 

1.26 

.50 

.84 

44.  Night  Vision 

1.88 

2.23 

.77 

.88 

1.93 

1.22 

.36 

.92 

45.  Peripheral  Vision 

2.02 

2.24 

.70 

.90 

2.03 

1.28 

.38 

.91 

46.  Depth  Perception 

1.73 

2.13 

.67 

.90 

1.91 

1.21 

.38 

.90 

47.  Glare  Sensitivity 

1.72 

2.07 

.74 

.87 

1.87 

1.13 

.42 

.86 

48.  Hearing  Sensitivity 

2.25 

2.14 

.91 

.82 

2.14 

1.21 

.56 

.78 

49.  Auditory  Attention 

3.17 

2.20 

1.39 

.60 

2.63 

1.24 

.81 

.57 

50.  Sound  Localization 

2.56 

2.34 

.88 

.86 

2.26 

1.25 

.48 

.85 

5 1 . Speech  Recognition 

4.10 

2.04 

1.46 

.49 

3.12 

1.21 

.88 

.47 

52.  Speech  Clarity 

4.34 

1.86 

1.21 

.57 

3.45 

1.12 

.08 

.46 

Note.  Statistics  are  based  on  33  occupations  with  Abilities  questionnaire  responses  from  at  least 
4 incumbents  (mean  number  of  incumbents  = 18.57,  median  = 13.0,  harmonic  mean  = 9.65). 

‘‘This  estimate  of  the  standard  error  of  measurement  was  calculated  as  SEM  = SD*  A/(  1 - rv) . 

bThis  estimate  of  reliability  was  obtained  by  calculating  the  intraclass  correlation  for  k ratings 
across  occupations:  ICC(l,k)  = IBMS-WMS1/BMS  (Shrout  & Fleiss,  1979),  where  k is  the 
harmonic  mean  of  the  number  of  ratings  provided  on  each  occupation. 


Table  9-2 

Reliability  of  Rated  Differences  Between  Occupations  Considering  Varying  Numbers  of 
Raters:  Abilities 


Descriptor 

1.  Oral  Comprehension 

2.  Written  Comprehension 

3.  Oral  Expression 

4.  ..Written  Expression 

5.  Fluency  of  Ideas 

6.  Originality 

7.  Problem  Sensitivity 

8.  Deductive  Reasoning 

9.  Inductive  Reasoning 

10.  Information  Ordering 

11.  Category  Flexibility 

12.  Math  Reasoning 

13.  Number  Facility 

14.  Memorization 

15.  Speed  of  Closure 

16.  Flexibility  of  Closure 

17.  Perceptual  Speed 

18.  Spatial  Orientation 

19.  Visualization 

20.  Selective  Attention 

21.  Timesharing 

22.  Arm-hand  Steadiness 

23.  Manual  Dexterity 

24.  Finger  Dexterity 

25.  Control  Precision 

26.  Multi-limb  Coordination 

27.  Response  Orientation 

28.  Rate  Control 

29.  Reaction  Time 

30.  Wrist-finger  Speed 

31.  Speed  of  Limb  Movmnt. 

32.  Static  Strength 

33.  Explosive  Strength 

34.  Dynamic  Strength 

35.  Trunk  Strength 

36.  Stamina 

37.  Extent  Flexibility 

38.  Dynamic  Flexibility 


Variable 

Level Importance 


II 

Do 

Ii 

Do 

.21 

.89 

.14 

.83 

.30 

.93 

.21 

.89 

.19 

.87 

.18 

.87 

.37 

.95 

.33 

.94 

.30 

.93 

.27 

.92 

.29 

.93 

.26 

.92 

.33 

.94 

.27 

.92 

.30 

.93 

.24 

.90 

.38 

.95 

.36 

.94 

.16 

.85 

.08 

.72 

.20 

.88 

.15 

.85 

.28 

.92 

.24 

.90 

.28 

.92 

.20 

.88 

.08 

.73 

.04 

.56 

.22 

.90 

.18 

.87 

.16 

.85 

.12 

.80 

.06 

.67 

.04 

.57 

.15 

.84 

.17 

.86 

.16 

.85 

.10 

.78 

.13 

.81 

.12 

.81 

.14 

.83 

.10 

.77 

.30 

.93 

.35 

.94 

.28 

.92 

.30 

.93 

.14 

.84 

.15 

.84 

.35 

.94 

.35 

.94 

.32 

.93 

.35 

.94 

.35 

.94 

.35 

.94 

.45 

.96 

.49 

.97 

.38 

.95 

.37 

.95 

.25 

.91 

.29 

.93 

.32 

.93 

.34 

.94 

.46 

.96 

.47 

.96 

.42 

.96 

.42 

.96 

.37 

.95 

.34 

.94 

.28 

.92 

.22 

.90 

.32 

.93 

.32 

.93 

.33 

.94 

.32 

.93 

.29 

.93 

.28 

.92 

Table  9-2  (continued) 


Reliability  of  Rated  Differences  Between  Occupations  Considering  Varying  Numbers  of 
Raters:  Abilities  — 


Descriptor 

39.  Gross  Body  Coord. 

40.  GrossBody  Equilib. 

41.  Near  Vision 

42.  Far  Vision 

* 43.  Visual  Color  Discrim 

44.  Night  Vision 

45.  Peripheral  Vision 

46.  Depth  Perception 

47.  Glare  Sensitivity 

48.  Hearing  Sensitivity 

49.  Auditory  Attention 

50.  Sound  Localization 

5 1 . Speech  Recognition 

52.  Speech  Clarity 


Variable 


Level 


Importance 


Il 

130 

Ei 

£30 

.34 

.94 

.31 

.93 

.36 

.94 

.36 

.94 

.14 

.83 

.08 

.73 

.28 

.92 

.26 

.92 

.31 

.93 

.31 

.93 

.38 

.95 

.48 

.97 

.44 

.96 

.47 

.96 

.44 

.96 

.44 

.96 

.37 

.95 

.35 

.94 

.28 

.92 

.24 

.90 

.11 

.79 

.10 

.77 

.34 

.94 

.33 

.94 

.07 

.71 

.07 

.69 

.10 

.78 

.07 

.69 

Note.  Reliability  estimates  are  based  on  a 33  occupations  with  Abilities  questionnaire 
responses  from  at  least  4 incumbents  (mean  number  of  incumbents  = 18.57,  median  = 
13.0,  harmonic  mean  = 9.65). 

aSingle  rater  estimates  of  reliability  were  obtained  by  calculating  the  intraclass  correlation 
for  single  judges  ratings  across  occupations:  ICC(1,1)  = fBMS- 

WMS]/[BMS+(k+ 1 )WMS]  (Strout  & Fleiss,  1979),  where  k is  the  harmonic  mean  of  the 
number  of  ratings  provided  on  each  occupation. 

‘‘Estimates  of  reliability  for  30  raters  were  obtained  by  applying  the  Spearman-Brown 
correction  formul;  to  the  single  rater  reliability  estimates. 


Table  9-3 

Reliability  of  Rated  Differences  Between  Occupations  Considering  Various  Recoding  Schemes: 
Abilities 


Type  of  Scale  and  Recoding  Applied 
Level Importance 


Descriptor 

Ea 

lb 

Ec 

Ea 

lb 

1.  Oral  Comprehension 

75 

79 

60 

66 

74 

2.  Written  Comprehension 

83 

85 

59 

75 

80 

3.  Oral  Expression 

73 

77 

55 

71 

80 

4.  Written  Expression 

87 

88 

67 

85 

87 

5.  Fluency  of  Ideas 

84 

83 

69 

81 

80 

6.  Originality 

83 

84 

58 

81 

84 

7.  Problem  Sensitivity 

85 

87 

50 

81 

85 

8.  Deductive  Reasoning 

83 

81 

70 

78 

79 

9.  Inductive  Reasoning 

88 

86 

71 

87 

88 

10.  Information  Ordering 

70 

68 

62 

50 

46 

11.  Category  Flexibility 

74 

75 

63 

68 

70 

12.  Math  Reasoning 

82 

81 

70 

78 

81 

13.  Number  Facility 

82 

85 

61 

74 

74 

14.  Memorization 

52 

60 

56 

33 

55 

15.  Speed  of  Closure 

77 

78 

73 

. 73 

77 

16.  Flexibility  of  Closure 

69 

72 

53 

61 

65 

17.  Perceptual  Speed 

45 

46 

31 

34 

00 

18.  Spatial  Orientation 

68 

71 

47 

71 

76 

19.  Visualization 

69 

72 

56 

58 

63 

20.  Selective  Attention 

63 

67 

65 

62 

67 

21.  Timesharing 

66 

71 

60 

57 

69 

22.  Arm-hand  Steadiness 

83 

82 

67 

86 

84 

23.  Manual  Dexterity 

82 

80 

69 

83 

82 

24.  Finger  Dexterity 

66 

62 

41 

67 

66 

25.  Control  Precision 

86 

85 

72 

86 

85 

26.  Multi-limb  Coord. 

85 

85 

74 

86 

86 

27.  Response  Orientation 

86 

87 

69 

86 

87 

28.  Rate  Control 

90 

91 

86 

92 

91 

29.  Reaction  Time 

88 

88 

76 

87 

85 

30.  Wrist-finger  Speed 

80 

75 

63 

83 

82 

31.  Speed  of  Limb  Mvmt. 

84 

84 

76 

85 

85 

32.  Static  Strength 

91 

91 

79 

91 

91 

33.  Explosive  Strength 

90 

90 

86 

89 

90 

34.  Dynamic  Strength 

87 

87 

81 

86 

86 

35.  Trunk  Strength 

82 

81 

65 

77 

80 

36.  Stamina 

85 

84 

75 

85 

83 

37.  Extent  Flexibility 

85 

85 

72 

85 

86 

38.  Dynamic  Flexibility 

83 

84 

78 

82 

83 

39.  Gross  Body  Coord. 

86 

84 

80 

84 

83 
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Table  9-3  (continued) 

Reliability  of  Rated  Differences  Between  Occupations  Considering  Various  Recoding  Schemes: 
Abilities 


Type  of  Scale  and  Recoding  Applied 
Level  Importance 

Pgggriptor n a rc  r,  r„ 


40. 

Gross-body  Equilib. 

87 

87 

85 

87 

±V 

87 

41. 

Near  Vision 

66 

65 

56 

51 

42 

42. 

Far  Vision 

82 

83 

60 

81 

81 

43. 

Visual  Color  Discrim. 

84 

80 

74 

84 

81 

44. 

Night  Vision 

88 

88 

81 

92 

92 

45. 

Peripheral  Vision 

90 

91 

81 

91 

90 

46. 

Depth  Perception 

90 

90 

84 

90 

89 

47. 

Glare  Sensitivity 

87 

87 

78 

86 

84 

48. 

Hearing  Sensitivity 

82 

82 

71 

78 

72 

49. 

Auditory  Attention 

60 

66 

45. 

57 

53 

50. 

Sound  Localization 

86 

86 

78 

85 

82 

51. 

Speech  Recognition 

49 

60 

41 

47 

68 

52. 

Speech  Clarity 

57 

63 

42 

46 

63 

Reliability  estimates  are  based  on  33  occupations  wit:  Abilities  questionnaire  responses 
from  at  least  4 incumbents  (mean  number  of  incumbents  = 18.57,  median  = 13.0,  harmonic  mean 
= 9.65).  Reliability  estimates  stipulated  as  were  calculated  using  the  full  eight  point  scale  for 
level,  and  retaining  all  of  the  data  for  the  importance  scale.  Reliability  estimates  stipulated  as  ij, 
were  calculated  using  a reduced  seven  point  scale  for  level,  and  excluding  the  data  for 
importance  scale  where  the  rater  marked  “NR”  on  the  level  scale.  Reliability  estimates  stipulated 
as  ic  were  calculated  using  binary  coded  scale  for  level  (relevant/not  relevant).  Decimals  are 
omitted. 


Table  9-4a 

Analysis  of  Variance  for  Descriptor.  Occupation,  and  Relevant  Interactions  as  Sources  of 
Variation  on  the  Level  Scale:  Abilities 


Source  of  Variation 

SS 

df 

MS 

F 

Occupations 

12416.76 

32 

388.02 

5.84* 

S (Occupations) 

38535.39 

580 

66.44 

Ability 

10016.13 

51 

196.39 

86.09* 

Ability  x Occupations 

20852.72 

1632 

12.78 

5.60* 

Ability  x S (Occupations) 

67481.18 

29580 

2.28 

Note.  Occupations  are  treated  as  random,  between-subjects  effects,  while  descriptors  are  treated 
as  fixed,  within  subjects  effects. 

*E<.05 


Table  9-4b 

Analysis  of  Variance  for  Descriptor,  Occupation,  and  Relevant  interactions  as  Sources  of 
Variation  on  the  Importance  Scale:  Abilities 


Source  of  Variation SS  df 


Occupations 

3316.19 

32 

103.63 

4.99* 

S (Occupations) 

12037.71 

580 

20.75 

Ability 

4159.78 

51 

81.56 

102.08* 

Ability  x Occupations 

7115.01 

1632 

4.36 

5.46* 

Ability  x S (Occupations) 

23634.86 

29580 

.80 

Note,  Occupations  are  treated  as  random,  between-subjects  effects,  while  descriptors  are  treated 
as  fixed,  within  subjects  effects. 

*P<.05 


Table  9-5 

Interrater  Agreement  Coefficients  for  Each  Scale  Type:  Abilities 


Number  of  Raters  on  Each  Variable 

Scale  Type & rt ^o_ 

Level  .82  .32  .93 

Importance ^2 .32 .93 

Note.  Interater  agreement  coefficient  estimates  are  based  on  33  occupations  with  Abilities 
questionnaire  responses  from  at  least  4 incumbents  (mean  number  of  incumbents  = 18.57, 
median  = 13.00,  harmonic  mean  = 9.65).  Full  sample  interrater  agreement  coefficients 
(rj  Were  obtained  by  considering  the  “Descriptor  x Occupations”  terms  from  tables  9-4a 
and  9-4b  as  true  variance.  Error  variance  was  defined  as  the  “Descriptor  x 
S (Occupations)”  term.  Estimates  for  reliability  for  1 and  30  raters  were  obtained  by 
applying  the  Spearman-Brown  correction  formula  to  the  k rater  reliability  estimates, 
where  k is  the  harmonic  mean  of  the  number  of  raters  for  each  occupation. 


Table  9-6a 

Analysis  of  Variance  for  Aggregate  Descriptor,  Occupation,  and  Relevant  Interactions  as  Sources 
of  Variation  on  the  Level  Scale:  Abilities 


Source  of  Variation 

SS 

df 

MS 

F 

Occupations 

3540.17 

32 

110.63 

5.69* 

S (Occupations) 

11273.01 

580 

19.44 

Aggregate 

2632.56 

14 

188.04 

142.35* 

Aggregate  x Occupations 

4229.49 

448 

9.44 

7.15* 

Aggregate  x S(Occupations) 

10726.37 

8120 

1.32 

Note.  Occupations  are  treated  as  random,  between-subjects  effects,  while  descriptors  are  treated 
as  fixed,  within  subjects  effects. 

*P<.05 


Table  9-6b 


* L — * "C vgi 

Sources  of  Variation  on  the  Importance  Scale:  Abilities 

df 

MS 

F 

Source  of  Variation 

SS 

Occupations 

3316.19 

32 

103.63 

4.99* 

S (Occupations) 

12037.71 

580 

20.75 

Aggregate 

4159.78 

51 

81.56 

102.08* 

Aggregate  x Occupations 

7115.01 

1632 

4.36 

5.46* 

Aggregate  x S(Occupations) 

23634.86 

29580 

.80 

Note.  Occupations  are  treated  as  random,  between-subjects  effects,  while  descriptors  are  treated 
as  fixed,  within  subjects  effects. 

*p<.05 
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Table  9-7 

Interrater  Agreement  Coefficients  for  Aggregate  Descriptors  for  Each  Scale  Tvne:  Abilities 


Scale  type 
Level 

Importance 

—Q^-  Interrater  agreement  coefficients  estimates  are  based  on  33  occupations  with  Abilities 
questionnaire  responses  from  at  least  4 incumbents  (mean  number  of  incumbents  = 18.57, 
median  — 13.0,  harmonic  mean  = 9.65).  Full  sample  interrater  agreement  coefficients  (rir)were 
obtained  by  considering  the  “Aggregate  x Occupations”  term  from  Tables  9-6a  and  9-6b  as  true 
variance.  Error  variance  was  defined  as  the  “Aggregate  x S (Occupations)”  term.  Estimates  of 
reliability  for  1 and  30  raters  were  obtained  by  applying  the  Spearman-Brown  correction  formula 
to  the  k rater  reliability  esitmates,  where  k is  the  harmonic  mean  of  the  number  of  raters  for  each 
occupation. 


Number  of  Raters  on  Each  Variable 
J* El £30 

.86  .39  .95 

J2 .32  .93 


Table  9-8 

Means  and  Standard  Deviations  of  Correlations  Between  the  Level  and  the  Importance  Scales 
Across  Occupations  and  Descriptors:  Abilities 

Level Importance 

Scale  na  M SD  n M SD 

Level 

Importance 

Note.  All  correlations  were  calculated  based  on  the  mean  of  ratings  assigned  by  raters  for  a given 
occupation,  descriptor,  and  scale.  Level-Importance  Means  above  the  diagonal  were  calculated 
by  taking  the  level  scale  means  on  a given  occupation  for  all  descriptors,  correlating  them  with 
importance  scale  means,  for  that  occupation,  and  then  averaging  them  with  the  correlations  for 
other  occupations.  Level-Importance  Means  below  the  diagonal  were  calculated  by  taking  the 
level  scale  means  for  a given  descriptor  for  all  occupations,  correlating  them  with  importance 
scale  means,  for  that  descriptor,  and  averaging  them  with  correlations  for  other  descriptors.  Other 
means  in  the  table  were  calculated  in  a similar  manner. 

aNumber  of  correlations  averaged,  not  number  of  observations  on  which  correlations  were 
calculated. 


33  .82  .30 


52  .92  .07 
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Table  9-9a 

Intercorrelations  of  Descriptors  for  the  Level  Scale  (Occupation-Level  Data):  Abilities 
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Table  9-9a  (continued) 

Intercorrelations  of  Descriptors  for  the  Level  Scale  (Occupation-Level  Data):  Abilities 
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Table  9-9a  (continued) 

Intercorrelations  of  Descriptors  for  the  Level  Scale  (Occupation-Level  DataV.  Abilities 
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Table  9-9a  (continued) 

Intercorrelations  of  Descriptors  for  the  Level  Scale  (Occupation-Level  Data):  Abilities 
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Table  9-9a  (continued) 

Intercorrelations  of  Descriptors  for  the  Level  Scale  (Occupation-Level  Data):  Abilities 
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Table  9-9a  (continued) 

Intercorrelations  of  Descriptors  for  the  Level  Scale  (Occupation-Lever Data):  Abilities 
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Table  9-9a  (continued) 

Intercorrelations  of  Descriptors  for  the  Level  Scale  (Occupation-Level  Data):  Abilities 
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Table  9-9a  (continued) 

Intercorrelations  of  Descriptors  for  the  Level  Scale  (Occupation-Level  Data):  Abilities 
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Table  9-9b  (continued) 

Intercorrelations  of  Descriptors  for  the  Importance  Scale  (Occupation-Level  Data):  Abilities 
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Table  9-9b  (continued) 

Intercorrelations  of  Descriptors  for  the  Importance  Scale  (Occupation-Level  Data):  Abilities 
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Table  9-9b  (continued) 

Intercorrelations  of  Descriptors  for  the  Importance  Scale  (Occupation-Level  Data):  Abilities 
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Table  9-9b  (continued) 

Intercorrelations  of  Descriptors  for  the  Importance  Scale  (Occupation-Level  Data):  Abilities 
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Table  9-9b  (continued) 

Intercorrelations  of  Descriptors  for  the  Importance  Scale  (Occupation-Level  Data):  Abilities 
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Table  9-9b  (continued) 

Intercorrelations  of  Descriptors  for  the  Importance  Scale  (Occupation-Level  Data):  Abilities 
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Table  9- 10a  (continued) 

Intercorrelations  of  Descriptors  for  the  Level  Scale  (Occupation-Level  Data):  Abilities 
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Table  9- 10a  (continued) 

Intercorrelations  of  Descriptors  for  the  Level  Scale  (Individual-Level  Data):  Abilities 
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Table  9- 10a  (continued) 

Intercorrelations  of  Descriptors  for  the  Level  Scale  (Individual-Level  Data):  Abilities 
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Table  9- 10a  (continued) 

Intercorrelations  of  Descriptors  for  the  Level  Scale  (Individual-Level  Data):  Abilities 
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Table  9- 10a  (continued) 

Intercorrelations  of  Descriptors  for  the  Level  Scale  (Individual-Level  Data):  Abilities 
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Table  9- 10a  (continued) 

Intercorrelations  of  Descriptors  for  the  Level  Scale  (Individual-Level  Data):  Abilities 
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Table  9- 10b 

Intercorrelations  of  Descriptors  for  the  Importance  Scale  (Occupation-Level  Data):  Abilities 
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Table  9- 10b  (continued) 

Intercorrelations  of  Descriptors  for  the  Importance  Scale  (Occupation-Level  Data):  Abilities 
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Table  9- 10b  (continued) 

Intercorrelations  of  Descriptors  for  the  Importance  Scale  f Occupation-Level  Data):  Abilities 
Descriptor  12  13  14  15  16  17  18  19  20  21 
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Table  9- 10b  (continued) 

Intercorrelations  of  Descriptors  for  the  Importance  Scale  (Occupation-Level  Data):  Abilities 
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Table  9- 10b  (continued) 

Intercorrelations  of  Descriptors  for  the  Importance  Scale  (Occupation-Level  Data):  Abilities 
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Table  9- 10b  (continued) 

Intercorrelations  of  Descriptors  for  the  Importance  Scale  (Occupation-Level  Data):  Abilities 
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Table  9- 10b  (continued) 

Intercorrelations  of  Descriptors  for  the  Importance  Scale  (Occunation-Level  Data):  Abilities 
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Table  9- 10b  (continued) 

Intercorrelations  of  Descriptors  for  the  Level  Scale  (Individual-Level  Data):  Abilities 
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Table  9-11 

Principal  Components  Analysis  Pattern  Matrix  for  the  Level  Scale:  Abilities  (loadings  over 


Factor 


Descriptor 

FI 

F2 

F3 

F4 

F5 

F6 

F7 

Communality 

1.  Oral  Comp. 

-.10 

.80 

.10 

-.30 

.27 

.08 

-.14 

.85 

2.  Written  Comp. 

-.13 

.89 

-.06 

.18 

-.05 

-.00 

-.09 

.86 

3.  Oral  Expression 

-.07 

.78 

.06 

.31 

.25 

-.17 

.06 

80 

4.  Written  Expression 

-.14 

.93 

-.18 

.09 

.06 

-.05 

-.06 

.93 

5.  Fluency  of  Ideas 

-.14 

.79 

.11 

-.12 

.35 

-.22 

.16 

.86 

6.  Originality 

.03 

.86 

.09 

.04 

.02 

-.27 

.23 

87 

7.  Problem  Sensitivity 

.16 

.79 

.09 

-.01 

.41 

-.00 

-.10 

• U / 

.83 

8.  Deductive  Reason. 

-.06 

.95 

.03 

.12 

-.11 

-.01 

-.08 

.94 

9.  Inductive  Reasoning 

.15 

.89 

-.00 

.07 

.09 

.06 

.04 

.84 

10.  Information  Ord. 

-.10 

.87 

.15 

.14 

-.15 

.23 

.12 

.89 

1 1.  Category  Hex. 

-.17 

.58 

.18 

.04 

.06 

-.02 

.58 

.73 

12.  Math  Reasoning 

-.22 

.76 

.30 

-.31 

-.21 

-.00 

-.14 

.87 

13.  Number  Facility 

-.32 

.66 

.35 

-.28 

-.13 

.11 

-.27 

83 

14.  Memorization 

-.06 

.63 

.10 

.52 

.20 

-.08 

-.16 

• UJ 

.75 

15.  Speed  of  Closure 

-.17 

.87 

-.04 

.02 

.13 

.24 

-.02 

.85 

16.  Flexibility  of  Clo. 

.21 

.64 

.40 

.12 

.18 

.12 

.31 

77 

17.  Perceptual  Speed 

.23 

.37 

.60 

.29 

-.13 

.48 

.23 

• / / 
.93 

18.  Spatial  Orientation 

.86 

.09 

-.13 

.12 

-.07 

-.01 

.24 

83 

19.  Visualization 

.24 

.38 

.71 

.06 

-.11 

-.17 

.10 

• OJ 

.77 

20.  Selective  Attention 

.16 

.54 

.14 

.52 

-.16 

.38 

.13 

.79 

21.  Time  Sharing 

.03 

.68 

.05 

.57 

.08 

.15 

.06 

.82 

22.  Arm-hand  Stead. 

.76 

.02 

.32 

.15 

.16 

.22 

-.08 

.78 

23.  Manual  Dexterity 

.81 

-.03 

.41 

.04 

-.02 

.26 

-.03 

89 

24.  Finger  Dexterity 

.60 

.04 

.60 

-.26 

-.00 

.29 

-.14 

89 

25.  Control  Precision 

.79 

.15 

.22 

-.10 

-.29 

.19 

-.27 

.07 

90 

26.  Multi-limb  Coord. 

.88 

-.15 

.03 

-.01 

-.09 

.18 

.11 

85 

27.  Response  Orient 

.94 

.08 

.05 

.02 

.09 

.11 

-.11 

• UJ 

.91 

28.  Rate  Control 

.92 

.04 

-.02 

.02 

-.16 

.02 

-.16 

90 

29.  Reaction  Time 

.92 

.05 

.04 

.02 

-.20 

.11 

-.15 

• ✓w 

.93 

30.  Wrist-finger  Spd. 

.06 

-.13 

-.04 

-.02 

.01 

.84 

.01 

73 

3 1 . Speed  of  Limb  Mv. 

.86 

-.27 

.01 

-.12 

.20 

.20 

-.05 

90 

32.  Static  Strength 

.88 

-.22 

.04 

-.05 

-.16 

-.06 

.16 

• ✓V 

89 

33.  Explosive  Strength 

.86 

-.09 

-.16 

-.15 

-.04 

-.07 

.30 

88 

34.  Dynamic  Strength 

35.  Trunk  Strength 

.88 

.83 

-.09 

-.29 

.10 

.17 

-.13 

-.23 

-.17 

.01 

-.07 

.10 

.31 

.31 

• OO 

.94 

95 

36.  Stamina 

.83 

-.23 

.14 

-.25 

.02 

-.06 

.28 

90 

37.  Extent  Hexibility 

.82 

-.38 

.26 

-.18 

-.05 

.13 

.09 

• ✓V 

93 

38.  Dynamic  Hex. 

.84 

-.40 

.11 

-.18 

.04 

.00 

.17 

93 

39.  Gross  Body  Coord.. 

.85 

-.32 

.24 

-.17 

.10 

-.02 

.17 

95 

40.  Gross-body  Equilib. 

.83 

-.22 

.10 

-.42 

-.06 

-.14 

.13 

95 

41.  Near  Vision 

.40 

.36 

.31 

.14 

.05 

.59 

-.15 

79 

42.  Far  Vision 

43.  Visual  Color  Disc. 

.75 

.66 

.11 

.12 

-.17 

.18 

.35 

.23 

.13 

.21 

-.12 

.26 

-.15 

-.04 

• / ✓ 
.78 
.64 

44.  Night  Vision 

.91 

-.02 

-.12 

.19 

.09 

.03 

-.04 

.89 

Table  9-11  (continued) 

Principal  Components  Analysis  Pattern  Matrix  for  the  Level  Scale:  Abilities  (loadings  over  .301 


Factor 


Descriptor 

FI 

F2 

F3 

F4 

F5 

F6 

F7 

Communality 

45.  Peripheral  Vision 

.90 

-.08 

-.03 

.28 

.08 

-.08 

-.02 

.90 

46.  Depth  Perception 

.95 

.10 

-.01 

.03 

-.14 

-.02 

-.11 

.94 

47.  Glare  Sensitivity 

.85 

.10 

-.22 

.09 

-.04 

.18 

-.31 

.92 

48.  Hearing  Sensitivity 

.84 

.05 

.23 

.25 

-.09 

-.07 

-.24 

.90 

49.  Auditory  Attention 

.74 

.13 

.28 

.29 

.11 

.04 

.06 

.74 

50.  Sound  Localization 

.89 

-.03 

.17 

.08 

.19 

-.01 

-.20 

.89 

51.  Speech  Recog. 

-.03 

.21 

-.03 

.02 

.87 

.03 

-.03 

.80 

52.  Speech  Clarity 

-.10 

.56 

-.15 

.08 

.73 

-.04 

.13 

.89 

Percent  of  Variance 

41 

25 

6 

5 

4 

3 

2 

Eigenvalue 

21.41 

13.20 

2.95 

2.45 

2.01 

1.69 

1.19 

Note.  N = 33.  The  correlation  matrix  was  based  on  means  calculated  at  the  occupation  level.  FI  = 
Psychomotor/  Perceptual,  F2  = Cognitive,  F3  = Visualization,  F4  = Time  Sharing  , F5  = Speech,  F6  = 
Wrist-Finger  Speed,  F7  = Non-interpretable.  These  loadings  are  based  on  an  orthogonal  varimax  rotation. 
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Table  9- 12a 

Descri ptor_Means  and  Standard  Deviations  on  the  Level  Scale  on  Six  Example  Jobs:  Abilities 
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Table  9- 1 2a  (continued) 

Descriptor  Means  and  Standard  Deviations  on  the  Level  Scale  on  Six  Example  Jobs:  Abilities 
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Table  9- 12a  (continued) 

Descriptor  Means  and  Standard  Deviations  on  the  Level  Scale  on  Six  Example  Jobs:  Abilities 
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Table  9- 12b 

Descriptor  Means  and  Standard  Deviations  on  the  Importance  Scale  on  Six  Example  Jobs:  Abilities 
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Table  9- 12b  (continued) 

Descriptor  Means  and  Standard  Deviations  on  the  Importance  Scale  on  Six  Example  Jobs:  Abilities 
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Table  9- 12b  (continued) 

Descriptor  Means  and  Standard  Deviations  on  the  Importance  Scale  on  Six  Example  Jobs:  Abilities 
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Table  9-13 

Rotated  Correlations  Between  Discriminating  Variables  and  Canonical  Discriminant  Functions 
for  Level  Scale:  Abilities 


Functions 


Descriptor 

FI 

F2 

F3 

F4 

F5 

IF2 

_2 

T1 

1 . Oral  Comprehension 

.03 

.39 

.14 

.00 

.12 

.19 

.31 

2.  Written  Comprehension 

-.09 

.55 

.05 

-.02 

.03 

.31 

.14 

3.  Oral  Expression 

.01 

.52 

.05 

.00 

.00 

.27 

.24 

4.  Written  Expression 

.00 

.62 

.05 

.06 

.12 

.40 

.09 

5.  Fluency  of  Ideas 

.10 

.54 

-.09 

.05 

.01 

.31 

.15 

6.  Originality 

.04 

.53 

-.05 

-.08 

.22 

.34 

.12 

7.  Problem  Sensitivity 

.13 

.50 

-.09 

.04 

.18 

.31 

.23 

8.  Deductive  Reasoning 

-.03 

.52 

-.03 

.12 

.17 

.32 

.14 

9.  Inductive  Reasoning 

.06 

.48 

-.05 

.22 

.20 

.32 

.14 

10.  Information  Ordering 

.08 

.28 

.12 

-.00 

.33 

.21 

.30 

11.  Category  Flexibility 

.11 

.29 

-.00 

-.01 

.15 

.12 

.25 

12.  Math  Reasoning 

.08 

.26 

-.02 

.05 

.59 

.43 

.34 

13.  Number  Facility 

.04 

.21 

.05 

.02 

.65 

.47 

.37 

14.  Memorization 

-.05 

.23 

.04 

.02 

.14 

.10 

.26 

15.  Speed  of  Closure 

.06 

.38 

.06 

-.03 

.16 

.18 

.28 

16.  Flexibility  of  Closure 

.24 

.30 

.06 

.01 

.17 

.18 

.11 

17.  Perceptual  Speed 

.16 

.09 

.08 

.16 

.15 

.10 

.09 

18.  Spatial  Orientation 

.24 

.05 

-.04 

.08 

.04 

.10 

.23 

19.  Visualization 

.24 

.20 

.02 

.07 

.17 

.13 

.24 

20.  Selective  Attention 

.01 

.14 

.03 

.01 

.05 

.01 

.12 

21.  Time  Sharing 

-.00 

.31 

.13 

.07 

.14 

.14 

.26 

22.  Arm-hand  Steadiness 

.41 

.13 

.05 

.14 

.06 

.21 

.26 

23.  Manual  Dexterity 

.42 

.05 

.08 

.24 

.04 

.24 

.10 

24.  Finger  Dexterity 

.41 

.06 

.16 

.23 

.14 

.27 

.19 

25.  Control  Precision 

.36 

-.02 

.05 

.46 

.24 

.40 

.26 

26.  Multi-limb  Coordination 

.55 

.09 

.17 

.19 

.13 

.39 

.15 

27.  Response  Orientation 

.43 

.12 

.16 

.48 

-.02 

.46 

.21 

28.  Rate  Control 

.43 

.08 

.02 

.49 

.04 

.43 

.30 

29.  Reaction  Time 

.37 

.05 

.09 

.50 

.00 

.40 

.36 

30.  Wrist-finger  Speed 

.18 

-.00 

.69 

.10 

.03 

.52 

.28 

31.  Speed  of  Limb  Movement 

.46 

.01 

.12 

.22 

-.00 

.27 

.23 

32.  Static  Strength 

.73 

.05 

.00 

.03 

.01 

.54 

.18 

33.  Explosive  Strength 

.70 

.10 

.09 

.09 

.02 

.52 

.24 

34.  Dynamic  Strength 

.66 

.04 

.00 

.13 

.05 

.46 

.16 

35.  Trunk  Strength 

.63 

-.00 

.09 

.07 

-.04 

.41 

.25 

36.  Stamina 

.60 

-.02 

-.00 

.10 

.06 

.37 

.24 

37.  Extent  Flexibility 

.62 

-.06 

.05 

.01 

.06 

.39 

.27 

38.  Dynamic  Flexibility 

.62 

-.02 

.09 

.02 

.09 

.40 

.24 

Table  9-13  (continued) 

Rotated  Correlations  Between  Discriminating  Variables  and  Canonical  Discriminant  Functions 
for  Level  Scale:  Abilities 


Functions 


Descriptor 

FI 

F2 

F3 

F4 

F5 

IE2 

2 

*1 

39.  Gross  Body  Coordination 

.60 

.00 

.03 

.15 

-.04 

.39 

.30 

40.  Gross-body  Equilibrium 

.64 

-.03 

-.06 

.05 

.06 

.42 

.30 

41.  Near  Vision 

.29 

.25 

.31 

.15 

.06 

.27 

.28 

42.  Far  Vision 

.28 

.15 

-.00 

.14 

.03 

.12 

.30 

43.  Visual  Color  Discrimination  .31 

.10 

-.04 

.14 

.06 

.13 

.14 

44.  Night  Vision 

.35 

.08 

-.01 

.29 

-.03 

.21 

.15 

45.  Peripheral  Vision 

.47 

.10 

.01 

.15 

.05 

.26 

.23 

46.  Depth  Perception 

.48 

.20 

.01 

.24 

.00 

.33 

.17 

47.  Glare  Sensitivity 

.29 

.13 

.08 

.28 

.02 

.19 

.23 

48.  Hearing  Sensitivity 

.30 

.06 

.04 

.22 

.11 

.16 

.29 

49.  Auditory  Attention 

.23 

.15 

.08 

.12 

-.00 

.10 

.23 

50.  Sound  Localization 

.32 

.02 

.06 

.27 

-.03 

.18 

.27 

51.  Speech  Recognition 

.03 

.10 

.08 

-.00 

.02 

.01 

.36 

52.  Speech  Clarity 

.02 

.24 

-.03 

.00 

.10 

.10 

.35 

Re 

.82 

.73 

.68 

.62 

.57 

Percent  of  Variance 

24 

13 

10 

7 

6 

Eigenvalue 

2.05 

1.14 

.87 

.63 

.50 

Note.  Statistics  are  based  on  33  occupations  with  Abilities  questionnaire  responses  from  at  least 
4 incumbents  (mean  number  of  incumbents  = 18.57,  median  = 13.0,  harmonic  mean  = 9.65).  FI 
= Physical,  F2  = Cognitive,  F3  = Wrist-Finger  Speed,  F4  = Psychomotor-Speed,  F5  = 
Mathematics.  Percent  of  "grouped"  cases  correctly  classified:  62.32%. 

XF2  = Sum  of  squared  rotated  correlations  across  five  functions. 

T|2  = Variance  in  Ability  Level  Scale  ratings  accounted  for  by  occupations. 
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Table  9- 14a 

Comparison  Between  Incumbent  and  Analyst  Descriptive  Statistics  Across  All  Occupations  and 
Reliability  Estimates  for  Rated  Differences  Between  Occupations  for  the  Level  Scale:  Abilities 


Descriptor 

Incumbent 

Analyst 

M 

SD 

Ik 

M 

SD 

Ik 

t 

F 

Eia 

d2 

1.  Oral  Comprehension 

4.31 

1.66 

.75 

3.94 

1.12 

.94 

4.13* 

2.20* 

.67 

.75 

2.  Written  Comprehen. 

4.29 

1.73 

.83 

3.72 

1.32 

.94 

5.71* 

1.72 

.76 

.94 

3.  Oral  Expression 

4.58 

1.69 

.73 

4.08 

1.20 

.95 

5.33* 

1.99 

.70 

.90 

4.  Written  Expression 

3.98 

1.85 

.87 

3.52 

1.46 

.95 

4.30* 

1.60 

.80 

.79 

5.  Fluency  of  Ideas 

2.77 

1.97 

.84 

2.88 

1.44 

.93 

-1.06 

1.89 

.82 

.43 

6.  Originality 

3.34 

1.99 

.83 

2.73 

1.42 

.93 

5.50* 

1.98 

.69 

1.39 

7.  Problem  Sensitivity 

4.21 

1.86 

.85 

3.20 

1.28 

.88 

9.89* 

2.09* 

.76 

1.83 

8.  Deductive  Reasoning 

3.64 

1.96 

.83 

3.25 

1.49 

.91 

3.48* 

1.73 

.83 

.85 

9.  Inductive  Reasoning 

3.37 

2.08 

.88 

3.02 

1.48 

.93 

2.97* 

1.96 

.77 

.97 

10.  Information  Ordering 

3.75 

1.87 

.70 

3.55 

1.09 

.82 

2.14* 

2.97* 

.62 

.56 

1 1 . Category  Flexibility 

2.46 

2.17 

.74 

2.95 

1.27 

.73 

-4.47* 

2.93* 

.37 

1.18 

12.  Math  Reasoning 

2.74 

2.01 

.82 

2.99 

1.36 

.92 

-2.30* 

2.19* 

.67 

.76 

13.  Number  Facility 

3.61 

1.86 

.82 

3.33 

1.26 

.93 

2.73* 

2.17* 

.67 

.83 

14.  Memorization 

4.06 

1.66 

.52 

3.49 

.99 

.85 

6.72* 

2.81* 

.59 

.67 

15.  Speed  of  Closure 

3.27 

1.92 

.77 

2.75 

1.50 

.82 

4.60* 

1.62 

.80 

.62 

16.  Flexibility  of  Closure 

2.45 

2.22 

.69 

2.28 

1.54 

.71 

1.45 

2.08* 

.58 

.81 

17.  Perceptual  Speed 

2.76 

2.08 

.45 

2.78 

1.33 

.69 

-.21 

2.44* 

.36 

.71 

1 8.  Spatial  Orientation 

2.68 

2.25 

.68 

2.44 

1.14 

.89 

2.24* 

3.90* 

.78 

.53 

19.  Visualization 

2.79 

2.11 

.69 

2.84 

1.42 

.91 

-.43 

2.20* 

.68 

.67 

20.  Selective  Attention 

3.98 

1.88 

.63 

2.67 

.99 

.73 

14.18* 

3.64* 

.41 

2.15 

21.  Time  Sharing 

4.10 

1.95 

.66 

2.74 

1.08 

.83 

13.86* 

3.26* 

.50 

2.25 

22.  Arm-hand  Steadiness 

2.26 

2.32 

.83 

2.50 

1.04 

.86 

-2.25* 

4.95* 

.58 

.91 

23.  Manual  Dexterity 

2.63 

2.29 

.82 

2.47 

1.10 

.87 

1.47 

4.28* 

.59 

1.26 

24.  Finger  Dexterity 

2.41 

2.20 

.66 

2.40 

1.20 

.80 

.09 

3.38* 

.50 

.92 

25.  Control  Precision 

2.13 

2.18 

.86 

2.23 

1.53 

.92 

-.83 

2.04* 

.85 

.62 

26.  Multi-limb  Coord. 

2.14 

2.25 

.85 

1.97 

1.40 

.89 

1.43 

2.59* 

.75 

.86 

27.  Response  Orientation 

2.50 

2.28 

.86 

2.34 

1.39 

.93 

1.36 

2.71* 

.86 

.69 

28.  Rate  Control 

1.75 

2.26 

.90 

1.27 

1.54 

.93 

3.89* 

2.15* 

.88 

1.15 

29.  Reaction  Time 

2.63 

2.63 

.88 

2.29 

1.69 

.94 

2.48* 

2.44* 

.87 

1.13 

30.  Wrist-finger  Speed 

3.50 

2.25 

.80 

2.84 

1.11 

.80 

6.08* 

4.11* 

.52 

.87 

3 1 . Speed  of  Limb  Mvmt.2.45 

2.43 

.84 

1.75 

1.30 

.85 

5.80* 

3.47* 

.74 

1.57 

32.  Static  Strength 

2.38 

2.32 

.91 

2.58 

1.60 

.95 

-1.57 

2.10* 

.80 

.79 

33.  Explosive  Strength 

1.52 

2.15 

.90 

1.51 

1.37 

.94 

.06 

2.46* 

.80 

.63 

34.  Dynamic  Strength 

1.61 

2.13 

.87 

1.37 

1.35 

.92 

2.08* 

2.49* 

.81 

.76 

35. Trunk  Strength 

2.14 

2.05 

.82 

2.50 

.84 

.91 

-3.76* 

5.97* 

.82 

.48 

36.  Stamina 

1.90 

2.03 

.85 

1.31 

1.12 

.89 

5.88* 

3.27* 

.79 

1.04 

37.  Extent  Flexibility 

2.52 

2.10 

.85 

2.61 

1.23 

.92 

-.78 

2.92* 

.70 

.74 

38.  Dynamic  Flexibility 

1.80 

2.01 

.83 

1.16 

1.10 

.81 

6.33* 

3.32* 

.76 

1.07 
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Table  9- 14a  (continued) 

Comparison  Between  Incumbent  and  Analyst  Descriptive  Statistics  Across  All  Occupations  and 
Reliability  Estimates  for  Rated  Differences  Between  Occupations  for  the  Level  Scale:  Abilities 


Descriptor 

Incumbent 

Analyst 

M 

SD 

Ek 

M 

SD 

Ik 

t 

F 

Da 

d2 

39.  Gross  Body  Coord. 

2.01 

2.03 

.86 

1.41 

1.03 

.86 

6.14* 

3.90* 

.77 

1.36 

40.  Gross-body  Equilib. 

1.68 

2.09 

.87 

1.23 

.96 

.90 

4.55* 

4.79* 

.80 

1.06 

41.  Near  Vision 

3.63 

1.92 

.66 

3.87 

.90 

.82 

-2.68* 

4.53* 

.37 

.73 

42.  Far  Vision 

2.52 

2.16 

.82 

2.47 

1.60 

.79 

.47 

1.83 

.75 

.53 

43.  Visual  Color  Discrim 

.2.21 

2.20 

.84 

2.43 

1.43 

.85 

-1.81 

2.37* 

.62 

.83 

44.  Night  Vision 

1.88 

2.23 

.88 

1.60 

1.39 

.86 

2.39* 

2.56* 

.84 

.69 

45.  Peripheral  Vision 

2.02 

2.24 

.90 

1.73 

1.25 

.90 

2.58* 

3.19* 

.87 

.91 

46.  Depth  Perception 

1.73 

2.13 

.90 

1.69 

1.30 

.93 

.36 

2.67* 

.87 

.50 

47.  Glare  Sensitivity 

1.72 

2.07 

.87 

1.26 

1.06 

.85 

4.51* 

3.81* 

.74 

1.14 

48.  Hearing  Sensitivity 

2.25 

2.14 

.82 

2.17 

1.23 

.87 

.71 

3.03* 

.71 

.80 

49.  Auditory  Attention 

3.17 

2.20 

.60 

2.42 

.92 

.81 

7.38* 

5.77* 

.55 

1.54 

50.  Sound  Localization 

2.56 

2.34 

.86 

1.83 

1.28 

.87 

6.28* 

3.36* 

.80 

1.82 

51.  Speech  Recognition 

4.10 

2.04 

.49 

2.95 

1.03 

.89 

11.61* 

3.95* 

.48 

1.68 

52.  Speech  Clarity 

4.34 

1.86 

.57 

3.23 

1.13 

.94 

11.46* 

2.72* 

.69 

1.75 

Note.  Incumbent  statistics  based  on  33  occupations  with  Ability  questionnaire  responses  from  at 
least  4 incumbents  (mean  number  of  incumbents  = 18.57,  median  =11,  harmonic  mean  = 9.65). 
Analyst  statistics  are  based  on  the  same  33  occupations  with  Ability  questionnaire  responses 
from  at  least  6 analysts  (mean  number  of  analysts  = 10.51,  median  = 12,  harmonic  mean  = 8.76). 
The  estimate  of  reliability  was  obtained  by  calculating  the  intraclass  correlation  for  k ratings 
across  occupations:  ICC(l.k)  = fBMS-WMSl/BMS  (Shrout  & Fleiss,  1979),  where  k is  the 
harmonic  mean  of  the  number  of  ratings  provided  on  each  occupation. 

The  t statistic  tests  for  differences  in  the  incumbent  and  analyst  group  means. 

The  F statistic  tests  for  differences  in  the  incumbent  and  analyst  group  standard  deviations. 

The  jja  correlation  indicates  the  degree  of  relationship  between  incumbent  and  analyst  mean 
occupations  ratings. 

The  d2  statistic  indicates  the  squared  differences  between  incumbent  and  analyst  mean 
occupations  ratings. 

*2  <.05 
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Table  9- 14b 

Comparison  Between  Incumbent  and  Analyst  Descriptive  Statistics  Across  All  Occupations  and 
Reliability  Estimates  for  Rated  Differences  Between  Occupations  for  the  Importance  Scale: 
Abilities 


Descriptor 

Incumbent 

Analyst 

M 

SD 

Ik 

M 

SD 

Ik 

t 

F 

Da 

d2 

1 . Oral  Comprehension 

3.70 

1.06 

.66 

3.95 

.96 

.86 

-3.85* 

1.22 

.52 

.36 

2.  Written  Comprehen. 

3.59 

1.06 

.75 

3.70 

1.11 

.86 

-1.51 

1.11 

.72 

.26 

3.  Oral  Expression 

3.71 

1.06 

.71 

4.04 

1.05 

.93 

-4.64* 

1.02 

.63 

.56 

4.  Written  Expression 

3.40 

1.11 

.85 

3.51 

1.21 

.91 

-1.39 

1.19 

.71 

.39 

5.  Fluency  of  Ideas 

2.51 

1.18 

.81 

2.68 

1.10 

.92 

-2.20* 

1.15 

.78 

.26 

6.  Originality 

2.72 

1.18 

.81 

2.48 

1.15 

.90 

3.04* 

1.05 

.70 

.50 

7.  Problem  Sensitivity 

3.41 

1.16 

.81 

3.28 

1.04 

.87 

1.69 

1.23 

.65 

.35 

8.  Deductive  Reasoning 

3.01 

1.17 

.78 

3.06 

1.06 

.89 

-.66 

1.22 

.75 

.24 

9.  Inductive  Reasoning 

2.79 

1.23 

.87 

2.96 

1.19 

.90 

-2.11* 

1.06 

.78 

.28 

10.  Information  Ordering 

3.19 

1.17 

.50 

3.72 

.80 

.73 

-8.24* 

2.10* 

.41 

.47 

1 1.  Category  Flexibility 

2.28 

1.23 

.68 

2.82 

.98 

.73 

-7.53* 

1.56 

.14 

.79 

12.  Math  Reasoning 

2.57 

1.23 

.78 

2.97 

1.12 

.92 

-5.06* 

1.20 

.66 

.43 

13.  Number  Facility 

3.08 

1.19 

.74 

3.34 

1.09 

.92 

-3.47* 

1.20 

.66 

.45 

14.  Memorization 

3.25 

1.05 

.33 

3.18 

.87 

.73 

1.04 

1.45 

.55 

.16 

15.  Speed  of  Closure 

2.71 

1.15 

.73 

2.52 

1.03 

.81 

2.66* 

1.25 

.76 

.20 

16.  Flexibility  of  Closure 

2.25 

1.25 

.61 

2.27 

1.09 

.73 

-.30 

1.32 

.60 

.23 

17.  Perceptual  Speed 

2.46 

1.22 

.34 

2.76 

1.04 

.77 

-4.04* 

1.36 

.34 

.45 

18.  Spatial  Orientation 

2.36 

1.27 

.71 

2.56 

1.08 

.88 

-2.60* 

1.37 

.70 

.29 

19.  Visualization 

2.42 

1.22 

.58 

2.78 

1.21 

.89 

-4.33* 

1.02 

.64 

.48 

20.  Selective  Attention 

3.23 

1.13 

.62 

2.86 

.84 

.64 

5.70* 

1.79 

.27 

.40 

21.  Time  Sharing 

3.16 

1.17 

.57 

2.71 

1.04 

.81 

6.13* 

1.27 

.46 

.53 

22.  Arm-hand  Steadiness 

2.12 

1.28 

.86 

2.68 

1.11 

.88 

-7.15* 

1.34 

.55 

.65 

23.  Manual  Dexterity 

2.31 

1.33 

.83 

2.86 

1.08 

.92 

-6.97* 

1.50 

.49 

.83 

24.  Finger  Dexterity 

2.21 

1.22 

.67 

2.67 

1.07 

.84 

-6.09* 

1.28 

.31 

.70 

25.  Control  Precision 

2.07 

1.21 

.86 

2.46 

1.32 

.92 

-4.46* 

1.18 

.80 

.41 

26.  Multi-limb  Coord. 

2.05 

1.23 

.86 

2.30 

1.20 

.90 

-3.09* 

1.06 

.72 

.39 

27.  Response  Orientation 

2.29 

1.32 

.86 

2.39 

1.15 

.91 

-1.24 

1.32 

.84 

.22 

28.  Rate  Control 

1.83 

1.19 

.92 

1.76 

1.16 

.95 

.95 

1.05 

.88 

.27 

29.  Reaction  Time 

2.37 

1.48 

.87 

2.27 

1.26 

.95 

1.06 

1.37 

.84 

.35 

30.  Wrist-finger  Speed 

2.78 

1.34 

.83 

2.84 

.97 

.82 

-.80 

1.92 

.57 

.37 

3 1 . Speed  of  Limb  Mvmt 

.2.17 

1.31 

.85 

1.99 

1.03 

.88 

2.33* 

1.63 

.80 

.28 

32.  Static  Strength 

2.14 

1.25 

.91 

2.47 

1.23 

.95 

-3.88* 

1.03 

.81 

.43 

33. Explosive  Strength 

1.70 

1.13 

.89 

1.81 

1.04 

.93 

-1.47 

1.16 

.74 

.29 

34.  Dynamic  Strength 

1.81 

1.16 

.86 

1.77 

1.01 

.93 

.62 

1.33 

.76 

.28 

35.  Trunk  Strength 

2.15 

1.20 

.77 

2.76 

.90 

.74 

-9.03* 

1.75 

.57 

.66 

36.  Stamina 

2.01 

1.19 

.85 

1.85 

1.05 

.86 

2.25* 

1.29 

.77 

.26 

37.  Extent  Flexibility 

2.25 

1.20 

.85 

2.62 

1.08 

.90 

-4.90* 

1.25 

.79 

.31 

38.  Dynamic  Flexibility 

1.90 

1.13 

.82 

1.59 

.84 

.86 

4.82* 

1.83 

.68 

.34 

39.  Gross  Body  Coord. 

2.04 

1.17 

.84 

1.80 

.92 

.87 

3.54* 

1.62 

.72 

.34 

1010 


Table  9- 14b  (continued) 

Comparison  Between  Incumbent  and  Analyst  Descriptive  Statistics  Across  All  Occupations  and 
Reliability  Estimates  for  Rated  Differences  Between  Occupations  for  the  Importance  Scale: 
Abilities 


Descriptor 

Incumbent 

Analyst 

M 

SD 

Ik 

M 

SD 

Ik 

t 

F 

lia 

d2 

40.  Gross  Body  Equilib. 

1.89 

1.19 

.87 

1.65 

.87 

.89 

3.52* 

1.87 

.79 

.31 

41.  Near  Vision 

2.90 

1.16 

.51 

3.80 

.94 

.58 

-13.12* 

1.51 

.29 

1.08 

42.  Far  Vision 

2.26 

1.19 

.81 

2.46 

1.16 

.88 

-2.48* 

1.04 

.77 

.27 

43.  Visual  Color  Discrim 

.2.16 

1.26 

.84 

2.45 

1.26 

.82 

-3.52* 

1.00 

.68 

.36 

44.  Night  Vision 

1.93 

1.22 

.92 

1.84 

1.09 

.91 

1.18 

1.25 

.88 

.20 

45.  Peripheral  Vision 

2.03 

1.28 

.91 

2.01 

1.15 

.88 

.27 

1.24 

.87 

.21 

46.  Depth  Perception 

1.91 

1.21 

.90 

2.04 

1.13 

.94 

-1.62 

1.16 

.85 

.23 

47.  Glare  Sensitivity 

1.87 

1.13 

.86 

1.60 

.97 

.90 

3.91* 

1.35 

.84 

.26 

48.  Hearing  Sensitivity 

2.14 

1.21 

.78 

2.29 

1.13 

.85 

-1.98* 

1.16 

.70 

.27 

49.  Auditory  Attention 

2.63 

1.24 

.57 

2.58 

.89 

.75 

.73 

1.92 

.21 

.51 

50.  Sound  Localization 

2.26 

1.25 

.85 

2.01 

1.05 

.85 

3.30* 

1.42 

.66 

.53 

5 1 . Speech  Recognition 

3.12 

1.21 

.47 

3.06 

1.01 

.78 

.77 

1.43 

.46 

.29 

52.  Speech  Clarity 

3.45 

1.12 

.46 

3.59 

1.05 

.91 

-1.93 

1.14 

.51 

.49 

Note.  Incumbent  statistics  based  on  33  occupations  with  Ability  questionnaire  responses  from  at 
least  4 incumbents  (mean  number  of  incumbents  = 18.57,  median  =11,  harmonic  mean  = 9.65). 
Analyst  statistics  are  based  on  the  same  33  occupations  with  Ability  questionnaire  responses 
from  at  least  6 analysts  (mean  number  of  analysts  = 10.51,  median  = 12,  harmonic  mean  = 8.76). 
The  estimate  of  reliability  was  obtained  by  calculating  the  intraclass  correlation  for  k ratings 
across  occupations:  ICC( l,k)  = rBMS-WMSl/BMS  (Shrout  & Fleiss,  1979),  where  k is  the 
harmonic  mean  of  the  number  of  ratings  provided  on  each  occupation. 

The  t statistic  tests  for  differences  in  the  incumbent  and  analyst  group  means. 

The  F statistic  tests  for  differences  in  the  incumbent  and  analyst  group  standard  deviations. 

The  ija  correlation  indicates  the  degree  of  relationship  between  incumbent  and  analyst  mean 
occupations  ratings. 

The  d2  statistic  indicates  the  squared  differences  between  incumbent  and  analyst  mean 
occupations  ratings. 

*£  < .05 


Table  9-15 

Comparison  of  Incumbent  and  Analyst  Descriptive  Statistics  Across  All  Occupations  for  the 
Level  Scale  Rescored  Dichotomouslv:  Abilities 


Incumbent Analyst 


Descriptor 

M 

SD 

Ik 

M 

SD 

Ik 

t 

1.  Oral  Comprehension 

.93 

.26 

.60 

1.00 

.00 



-5.17* 

2.  Written  Comprehension 

.93 

.26 

.59 

1.00 

.05 

— 

-4.91* 

3.  Oral  Expression 

.93 

.25 

.55 

1.00 

.00 

— 

-4.98* 

4.  Written  Expression 

.92 

.27 

.67 

.99 

.08 

— 

-5.05* 

5.  Fluency  of  Ideas 

.76 

.43 

.69 

.95 

.23 

.42 

-7.59* 

6.  Originality 

.82 

.38 

.58 

.95 

.22 

.44 

-5.77* 

7.  Problem  Sensitivity 

.92 

.27 

.50 

1.00 

.05 

— 

-5.10* 

8.  Deductive  Reasoning 

.85 

.36 

.70 

.97 

.16 

.56 

-6.08* 

9.  Inductive  Reasoning 

.82 

.39 

.71 

.98 

.15 

.58 

-7.43* 

10.  Information  Ordering 

.89 

.31 

.62 

.99 

.08 

.14 

-5.96* 

11.  Category  Flexibility 

.63 

.48 

.63 

.95 

.23 

— 

-11.36* 

12.  Math  Reasoning 

.75 

.43 

.70 

.96 

.19 

.02 

-8.71* 

jl3.  Number  Facility 

.89 

.31 

.61 

1.00 

.05 

— 

-6.18* 

14.  Memorization 

.93 

.26 

.56 

1.00 

.05 



-5.17* 

15.  Speed  of  Closure 

.81 

.39 

.73 

.87 

.34 

.27 

-2.20* 

16.  Flexibility  of  Closure 

.63 

.48 

.53 

.81 

.40 

.37 

-5.92 

17.  Perceptual  Speed 

.72 

.45 

.31 

.95 

.23 

.22 

-8.79* 

18.  Spatial  Orientation 

.68 

.47 

.47 

.95 

.21 

.66 

-10.29* 

19.  Visualization 

.74 

.44 

.56 

.94 

.24 

.39 

-7.71* 

20.  Selective  Attention 

.92 

.27 

.65 

.99 

.08 

-4.85* 

21.  Timesharing 

.89 

.31 

.60 

.99 

.11 



-5.71* 

22.  Arm-hand  Steadiness 

.57 

.49 

.67 

.99 

.12 

__ 

-15.23* 

23.  Manual  Dexterity 

.66 

.47 

.69 

.99 

.12 

«... 

-12.55* 

24.  Finger  Dexterity 

.64 

.48 

.41 

.93 

.25 

.73 

-10.44* 

25.  Control  Precision 

.59 

.49 

.72 

.84 

.37 

.57 

-8.12* 

26.  Multi-limb  Coordination 

.57 

.50 

.74 

.83 

.38 

.65 

-8.40* 

27.  Response  Orientation 

.65 

.48 

.69 

.93 

.26 

.58 

-9.83* 

28.  Rate  Control 

.44 

.50 

.86 

.52 

.50 

.80 

-2.40* 

29.  Reaction  Time 

.59 

.49 

.76 

.86 

.35 

.62 

-8.80* 

30.  Wrist-finger  Speed 

.80 

.40 

.63 

.97 

.16 

.12 

- 7.63* 

3 1.  Speed  of  Limb  Movement 

.60 

.49 

.76 

.79 

.41 

.61 

-6.24* 

32.  Static  Strength 

.63 

.48 

.79 

.95 

.21 

.60 

-11.73* 

33.  Explosive  Strength 

.42 

.49 

.86 

.69 

.46 

.80 

-8.31* 

34.  Dynamic  Strength 

.45 

.50 

.81 

.66 

.48 

.72 

-6.33* 

35.  Trunk  Strength 

.66 

.47 

.65 

.99 

.12 

.22 

-12.60* 

36;  Stamina 

.59 

.49 

.75 

.77 

.42 

.70 

-5.84* 

37.  Extent  Flexibility 

.73 

.45 

.72 

.97 

.18 

.28 

-9.70* 

38.  Dynamix  Flexibility 

.56 

.50 

.78 

.63 

.48 

.58 

-2.25* 

39.  Gross  Body  Coordination 

.64 

.48 

.80 

.76 

.43 

.69 

-4.01* 

Table  9-15  (continued) 

Comparison  of  Incumbent  and  Analyst  Descriptive  Statistics  Across  AH  Occupations  for  the 
Level  Scale  Rescored  Dichotomouslv:  Abilities 


Incumbent Analyst 


Descriptor 

M 

SD 

Dc 

M 

SD 

Dc 

t 

40. 

Gross  Body  Equilibrium 

.49 

.50 

.85 

.74 

.44 

.76 

-7.56* 

41. 

Near  Vision 

.85 

.35 

.56 

1.00 

.00 

— 

-7.72* 

42. 

Far  Vision 

.68 

.47 

.60 

.83 

.37 

.62 

-5.22* 

43. 

Visual  Color  Discrimination 

.62 

.49 

.74 

.88 

.32 

.53 

-9.20* 

44. 

Night  Vision 

.49 

.50 

.81 

.73 

.45 

.66 

-7.27* 

45. 

Peripheral  Vision 

.55 

.50 

.81 

.84 

.37 

.71 

-9.40* 

46. 

Depth  Perception 

.49 

.50 

.84 

.80 

.40 

.78 

-10.16* 

47. 

Glare  Sensitivity 

.51 

.50 

.78 

.79 

.41 

.50 

-8.96* 

48. 

Hearing  Sensitivity 

.62 

.49 

.71 

.91 

.28 

.23 

-10.27* 

49. 

Auditory  Attention 

.79 

.41 

.45 

.99 

.11 

.01 

-8.96* 

50. 

Sound  Localization 

.66 

.47 

.78 

.85 

.36 

.46 

-6.47* 

51. 

Speech  Recognition 

.90 

.30 

.41 

1.00 

.05 

— 

-5.84* 

52. 

Speech  Clarity 

.94 

.24 

.42 

1.00 

.00 

— 

-4.65* 

Note.  Incumbent  statistics  are  based  on  35  occupations  with  Skills  questionnaire  responses  from 
at  least  4 incumbents  (mean  number  of  incumbents  = 18.51,  median  = 12,  harmonic  mean  = 
9.01).  Analyst  statistics  are  based  on  the  same  35  occupations  with  Skills  questionnaire  responses 
from  at  least  6 analysts  (mean  number  of  analysts  = 10.29,  median  = 12.0,  harmonic  mean  = 
8.66). 

The  estimate  of  reliability  was  obtained  by  calculating  the  intraclass  correlation  for  k ratings 
across  occupations:  ICC(l,k)  = fBMS-WMSl/BMS  (Shrout  & Fleiss,  1979),  where  k is  the 
harmonic  mean  of  the  number  of  ratings  provided  on  each  occupation. 

The  t statistic  tests  for  differences  in  the  incumbent  and  analyst  group  means. 

*P<.05 
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Table  9-16 

Principal  Components  Analysis  Pattern  Matrix  for  the  Analyst  Level  Scale:  Abilities 


Factor 


Descriptor 

FI 

F2 

F3 

F4 

F5 

F6 

Communality 

1.  Oral  Comp. 

.93 

-.08 

-.20 

-.06 

-.01 

-.18 

.95 

2.  Written  Comp. 

.92 

-.16 

-.26 

-.14 

.05 

-.02 

.96 

3.  Oral  Expression 

.88 

-.13 

-.27 

. -.12 

-.12 

-.24 

.95 

4.  Written  Expression 

.90 

-.18 

-.33 

-.17 

.01 

-.02 

.97 

5.  Fluency  of  Ideas 

.93 

-.05 

-.02 

.07 

-.28 

-.00 

.94 

6.  Originality 

.91 

-.02 

.01 

.06 

-.29 

.01 

.92 

7.°Problem  Sensitivity 

.91 

.19 

-.04 

.14 

-.02 

-.03 

.89 

8.  Deductive  Reason. 

.95 

-.02 

-.14 

.03 

.07 

.17 

.95 

9.  Inductive  Reasoning 

.95 

.12 

-.06 

.04 

-.03 

.06 

.93 

10.  Inductive  Reasoning 

.77 

-.07 

.30 

.19 

.40 

.15 

.91 

1 1 . Information  Ord. 

.84 

-.25 

-.00 

.13 

-.08 

-.09 

.79 

12.  Category  Flex. 

.86 

-.28 

-.24 

-.04 

.18 

.13 

.92 

13.  Math  Reasoning 

.77 

-.35 

-.26 

-.12 

.19 

.08 

.84 

14.  Number  Facility 

.80 

.01 

-.14 

.01 

-.24 

-.40 

.87 

15.  Memorization 

.86 

.11 

-.17 

.10 

.26 

-.16 

.88 

16.  Speed  of  Closure 

.67 

.51 

.21 

.05 

.04 

.23 

.80 

17.  Perceptual  Speed 

.34 

-.00 

-.04 

.29 

.74 

.09 

.76 

18.  Perceptual  Speed 

-.10 

.79 

.40 

.17 

-.17 

-.08 

.86 

19.  Visualization 

.45 

.18 

-.56 

.47 

-.01 

.27 

.93 

20.  Selective  Attention 

.61 

.57 

-.10 

.35 

.17 

-.05 

.87 

21.  Time  Sharing 

.47 

.65 

-.11 

-.04 

-.28 

-.17 

.76 

22.  Arm-Hand  Stead. 

.09 

.28 

.60 

.62 

.24 

-.15 

.91 

23.  Manual  Dexterity 

-.19 

.25 

.76 

.43 

.30 

.05 

.95 

24.  Finger  Dexterity 

.01 

.13 

.35 

.63 

.60 

.10 

.91 

25.  Control  Precision 

-.06 

.61 

.30 

.52 

.34 

.27 

.93 

26.  Multi-limb  Coord. 

-.28 

.69 

.55 

.19 

.11 

.11 

.91 

27.  Response  Orient. 

-.17 

.92 

.18 

.08 

.11 

.03 

.93 

28.  Rate  Control 

-.13 

.91 

.22 

.01 

.04 

.17 

.91 

29.  Reaction  Time 

-.14 

.87 

.31 

.27 

.12 

.01 

.95 

30.  Wrist-Finger  Speed 

.23 

-.07 

.07 

-.04 

.81 

.01 

.72 

3 1 . Speed  of  Limb  Mv. 

-.20 

.71 

.57 

.04 

.16 

-.09 

.90 

32.  Static  Strength 

-.23 

.48 

.75 

.19 

.06 

-.05 

.89 

33.  Explosive  Strength 

-.09 

.70 

.65 

.07 

-.02 

.17 

.95 

34.  Dynamic  Strength 

-.24 

.51 

.77 

.14 

.09 

.07 

.94 

35.  Trunk  Strength 

-.37 

.35 

.80 

.21 

.03 

.11 

.96 

36.  Stamina 

-.11 

.55 

.73 

-.17 

-.14 

.00 

.90 

37.  Extent  of  Flexibility 

-.28 

.30 

.83 

.28 

.03 

.11 

.95 

38.  Dynamic  Flexibility 

-.37 

.34 

.77 

.16 

-.04 

.10 

.88 

39.  Gross  Body  Coord. 

-.26 

.65 

.59 

-.01 

-.19 

-.04 

.89 

40.  Gross  Body  Equil. 

-.18 

.45 

.82 

.10 

-.06 

.04 

.90 

O 

ERLC 


Table  9-16  (continued) 

Principal  Components  Analysis  Pattern  Matrix  for  the  Analyst  Level  Scale:  Abilities 


Factor 


Descriptor 

FI 

F2 

F3 

F4 

F5 

F6 

Communality 

41.  Near  Vision 

.74 

-.16 

-.36 

.02 

.43 

-.11 

.90 

42.  Far  Vision 

.24 

.74 

.17 

-.01 

-.45 

.03 

.86 

43.  Visual  Color  Discrim. 

.09 

.32 

.40 

.76 

-.04 

.11 

.86 

44.  Night  Vision 

.11 

.82 

.37 

.00 

-.16 

-.25 

.91 

45.  Peripheral  Vision 

-.15 

.88 

.27 

.10 

-.18 

-.06 

.91 

46.  Depth  Perception 

-.15 

.74 

.52 

.16 

.01 

.26 

.93 

47.  Glare  Sensitivity 

.04 

.86 

.21 

-.05 

.09 

.20 

.83 

48.  Hearing  Sensitivity 

-.03 

.57 

.42 

.50 

.16 

.23 

.84 

49.  Auditory  Attention 

.14 

.64 

.18 

.44 

.03 

-.04 

.66 

50.  Sound  Localization 

-.07 

.85 

.18 

.36 

.05 

-.05 

.89 

51.  Speech  Recognition 

.46 

-.14 

-.33 

-.31 

-.11 

-.65 

.87 

52.  Speech  Clarity 

.75 

-.00 

-.34 

-.26 

-.26 

-.36 

.93 

Percent  of  Variance 

44 

26 

9 

5 

2 

2 

Eigenvalue 

23.07 

13.72 

4.80 

2.47 

1.23 

1.12 

Note.  The  correlation  matrix  was  based  on  means  calculated  at  the  occupation  level.  FI  = 
Cognitive,  F2  = Psychomotor/Perceptual,  F3  = Physical,  F4  = Sensory-Dexterity,  F5  = Wrist- 
Finger  Speed,  F6  = Speech.  These  loadings  are  based  on  an  orthogonal  varimax  rotation. 


Figure  9-1 

Descriptions  and  Definitions  of  Abilities 
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Figure  9-1  (continued) 

Descriptions  and  Definitions  of  Abilities 
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Figure  9-1  (continued) 

Descriptions  and  Definitions  of  Abilities 
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Figure  9-1  (continued) 

Descriptions  and  Definitions  of  Abilities 
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Figure  9- 1 (continued) 

Descriptions  and  Definitions  of  Abilities 
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Figure  9-1  (continued) 

Descriptions  and  Definitions  of  Abilities 
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Figure  9-1  (continued) 

Descriptions  and  Definitions  of  Abilities 
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Figure  9-1  (continued)’ 

Descriptions  and  Definitions  of  Abilities 
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Figure  9-1  (continued) 

Descriptions  and  Definitions  of  Abilities 
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Figure  9-1  (continued) 

Descriptions  and  Definitions  of  Abilities 
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Figure  9-1  (continued) 

Descriptions  and  Definitions  of  Abilities 
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Figure  9-1  (continued) 

Descriptions  and  Definitions  of  Abilities 
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Figure  9-2 

Example  Page  from  Abilities  Questionnaire 


1.  Oral 

Comprehension 


The  ability  to  listen  to  and  understand  information  and  ideas 
presented  through  spoken  words  and  sentences. 


Level 

What  level  of  this  ability  is  needed  to  perform  this  job? 


Requires  understanding  complex  or 
detailed  spoken  sentences  that  contain 
unusual  words  and  phrases. 


Requires  understanding  short  or  simple 
spoken  sentences  that  contain  common 
words  and  phrases. 


©-- 

@» 

©-- 

©-- 
©"  . 

©*L 

Not  relevant  at  all  for  performance  on  this  job 


Understanding  a lecture  on  advanced 
physics. 


Understanding  a coach *s  oral  instructions 
for  a sport 


Understanding  a television  commercial. 


Importance 

How  important  is  this  ability  to  performance  on  this  job? 

Not 

Important 

Somewhat 

Important 

Important 

Very 

Important 

Extremely 

Important 

© 

© 

© 

© 

© 

1 


Figure  9-3 

Thirty-Three  Occupations  With  Four  or  More  Incumbents  Completing  the  Abilities 
Questionnaire 


Occupation 

Code 

Occupation  Title 

Number  of 
Respondents 

15005 

Education  Administrators 

8 

19005 

General  Managers  & Top  Executives 

43 

21108 

Loan  Officers  & Counselors 

5 

22135 

Mechanical  Engineers 

r 13 

25105 

Computer  Programmers 

6 

27311 

Recreation  Workers 

4 

31303 

Teachers,  Preschool 

6 

31305 

Teachers,  Elementary  School 

13 

32502 

Registered  Nurses 

30 

32902 

Medical  & Clinical  Laboratory  Technologists 

8 

49008 

Salespersons,  Except  Scientific  & Retail 

7 

49011 

Salespersons,  Retail 

21 

49021 

Stock  Clerks,  Sales  Floor 

8 

49023 

Cashiers 

16 

51002 

First  Line  Supervisors,  Clerical/Administrative 

47 

53311 

Insurance  Claims  Clerks 

7 

55108 

Secretaries, 

63 

55305 

Receptionists  & Information  Clerks 

5 

55338 

Bookkeeping,  Accounting,  & Auditing  Clerks 

28 

55347 

General  Office  Clerks 

76 

j 61005 

Police  & Detective  Supervisors 

13 

63014 

Police  Patrol  Officers 

23 

65008 

Waiters  & Waitresses 

19 

65038 

Food  Preparation  Workers 

16 

66008 

Nursing  Aides,  Orderlies,  & Attendants 

12 

67005 

Janitors  & Cleaners 

27 

85132 

Maintenance  Repairers,  General  Utility 

38 

87902 

Earth  Drillers 

4 

89108 

Machinists 

4 

| 92974 

Packaging  & Filling  Machine  Operators 

11 

97102 

Truck  Drivers 

16 

97111  I 

Bus  Drivers,  Schools 

12 

Chapter  10 

Occupational  Interests  and  Values: 

Evidence  for  the  Reliability  and  Validity 
of  the  Occupational  Interest  Codes  and  the  Values  Measures 

Christopher  E.  Sager 
American  Institutes  for  Research 


The  primary  motivation  for  describing  people  and  occupations  in  terms  of  vocational 
interests  and  values  is  to  facilitate  the  match  between  the  two.  As  Sager  (1995)  states: 

It  is  an  enduring  proposition  that  performance  is  a function  of  ability  and 
motivation.  As  pointed  out  by  Hakel  (1986)  and  Dawis  (1991),  understanding 
interests  and  values  is  part  of  understanding  motivation.  The  idea  is  that 
individuals  who  are  motivated  will  perform  well,  and  that  interests  and  values  are 
important  parts  of  motivation.  Maximizing  performance,  however,  is  not  the  only 
reason  for  trying  to  achieve  good  matches  between  people  and  jobs.  Borgen, 

Weiss,  Tinsley,  Dawis,  and  Lofquist  (1968)  point  to  the  theory  that  satisfaction  is 
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dependent  on  the  extent  to  which  the  job  matches  a person’s  interests  and  values. 
Therefore,  if  the  goals  of  0*NET  include  the  description  of  occupations  for  the 
purpose  of  person-job  matching,  then  occupational  interests  and  values  are 
potentially  an  important  part  of  the  content  model,  (p.  1 1-1). 

This  chapter  first  briefly  reviews  the  taxonomies  of  the  occupational  interests  and  values 
descriptors  that  are  part  of  the  0*NET  content  model,  second,  discusses  the  reliability  and 
validity  evidence  related  to  the  measures  of  these  descriptors  used  in  the  initial  0*NET  data 
collection  effort,  and,  finally,  discusses  the  implications  of  the  reliability  and  validity  evidence 
for  potential  revisions  to  the  content  model  and  measures. 

The  0*NET  occupational  interests  and  values  measures  and  their  development  are 
discussed  in  Sager  (1995).  In  the  occupational  domain  interests  and  values  are  defined  as 
relatively  stable  dispositions,  developed  over  time,  that  are  based  on  affective  evaluations  of 
life’s  experiences  (Dawis,  1991).  Interests  are  generally  viewed  as  the  relative  like  or  dislike  of 
particular  activities  while  occupational  values  are  viewed  as  judgments  about  the  importance  of 
activities  or  other  characteristics  of  the  work  environment.  For  example,  the  Strong  Interest 
Inventory  (SII)  is  an  interest  measure;  some  of  its  items  ask  people  to  indicate  whether  they  like, 
are  indifferent  towards,  or  dislike  particular  work  and  leisure  activities  (Hansen  & Campbell, 
1985).  The  Minnesota  Information  Questionnaire  (MIQ)  is  a values  measure;  its  items  require 
people  to  evaluate  the  relative  importance  of  different  characteristics  of  their  “ideal”  job,  such  as 
Security  (Descriptor  #14)  and  Variety  (Descriptor  #19)  (Dawis  & Lofquist,  1984). 

In  addition  to  the  conceptual  distinction  between  occupational  interests  and  values,  the 
empirical  evidence  suggests  that  individual  difference  measures  of  these  constructs  have 
complimentary  strengths  and  weaknesses  (Dawis,  1991).  Research  shows  that  values 
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discriminate  among  occupations;  however,  there  is  relatively  more  research  supporting  the 
hypothesis  that  interest  measures  predict  later  occupational  membership.  While  values  appear  to 
be  relatively  strong  predictors  of  future  job  satisfaction  the  evidence  for  the  influence  of  interests 
on  future  job  satisfaction  is  mixed.  Because  occupational  interests  and  values  are  theoretically 
different  and  their  measures  show  different  patterns  of  relationships  with  external  variables  (e.g., 
future  occupational  membership  and  job  satisfaction),  it  was  determined  that  the  prototype  of  the 
0*NET  content  model  would  include  occupation  descriptors  in  both  domains  (Sager,  1995). 

Interests 

Taxonomy 

Holland’s  six  factor  taxonomy  of  occupational  interests  was  chosen  to  define  interests 
within  the  0*NET  content  model  (Sager,  1995).  Holland’s  six  factor  taxonomy  is  used  to 
describe  people  and  occupations  (Gottfredson  & Holland,  1989;  Holland,  1976).  Figure  10-1 
contains  the  titles  and  descriptions  of  the  Strong  Interest  Inventory  General  Occupational  Themes 
(i.e.,  Holland’s  six  types)  that  are  used  to  describe  people.  Here,  Holland  codes  describe  an 
occupation  in  terms  of  the  three  Holland  types  that  best  fit  the  occupation  (i.e.,  High-point 
codes).  An  occupation’s  code  consists  of  the  first  letter  of  each  of  the  three  relevant  types  (i.e.,  R 
= Realistic;  I = Investigative;  A = Artistic;  S = Social;  E = Enterprising;  and  C = Conventional) 
presented  in  order  of  importance.  For  example,  the  Holland  code  for  the  0*NET  occupation 
Education  Administrators  is  ESA,  meaning  that  this  occupation  is  primarily  described  as  an 
Enterprising  occupation,  secondarily  as  a Social  occupation,  and  as  having  some  Artistic 
elements  (Gottfredson  & Holland,  in  press). 
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Procedures 

The  Holland  codes  for  6 of  the  occupations  that  are  part  of  the  0*NET  prototype  are 
presented  in  Figure  10-2  with  their  0*NET  occupation  codes  and  titles:  (1)  General  Managers 
and  Top  Executives,  (2)  Computer  Programmers,  (3)  Registered  Nurses,  (4)  Police  Patrol 
Officers,  (5)  Janitors  and  Cleaners,  Except  Maids  and  Housekeeping,  and  (6)  Maintenance 
Repairers,  General  Utility. 

The  codes  are  reproduced  here  with  permission  from  Psychological  Assessment 
Resources,  Incorporated  from  the  Dictionary  of  Holland  Occupational  Codes  fDHOO.  Third 
Edition  (Gottffedson  & Holland,  in  press).  The  Holland  codes  for  80  of  the  0*NET  occupations 
were  derived  from  Holland  codes  generated  for  their  related  Dictionary  of  Occupational  Titles 
(DOT)  occupations.1  The  process  is  briefly  summarized  here. 

First,  multiple  discriminant  analysis  was  used  to  develop  classification  functions  based,  on 
193  DOT  occupations  for  which  there  was  confidence  regarding  their  Holland  codes.  These 
functions  used  DOT  job  analysis  data  to  estimate  the  probability  of  assignment  of  each 
occupation  to  each  Holland  type.  Each  occupation  was  assigned  a three-letter  code  such  that  the 
letters  represented  the  most  probably  types  in  descending  order.  The  functions  were  then  used  to 
generate  codes  for  12,741  DOT  occupations.  The  new  codes  for  these  occupations  were  reviewed 
for  consistency  with  other  job  analysis  information  including  their  associated  codes  from  the 
second  edition  of  the  DHOC  (Gottffedson  & Holland,  1989).  This  review  resulted  in  some 
adjustments;  however,  there  were  veiy  few  changes  in  Holland  codes  for  DOT  occupations 
between  the  second  and  third  editions  of  the  DHOC  (i.e.,  only  3%  of  these  DOT  occupations 
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In  the  third  edition  of  the  DHOC  (Gottffedson  & Holland,  in  press)  the  Holland  codes  for 
the  0*NET  occupations  are  derived  from  the  codes  for  their  associated  DOT  occupations.  As 
indicated  earlier  the  0*NET  occupational  structure  is  based  on  the  Occupational  Employment 
Statistics  (OES)  structure,  and  the  DOT  occupations  have  been  linked  to  OES  occupations 
(National  Crosswalk  Services  Center,  1993).  Because  there  are  many  more  occupations  in  the 
DOT  structure  than  the  0ES/0*NET  structure,  the  linkages  are  generally  multiple  DOT 
occupations  to  one  0ES/0*NET  occupation.  Within  each  0ES/0*NET  occupation  scores  were 
assigned  to  the  Holland  types  for  each  DOT  occupation;  a score  of  3 was  assigned  to  the  type  if  it 
was  the  first  in  the  occupation’s  code,  2 if  it  was  the  second,  1 if  it  was  the  third,  and  0 if  it  was 
not  in  the  top  three.  The  scores  for  each  type  were  summed  across  the  DOT  occupations  linked  to 
each  0ES/0*NET  occupation.  The  three  Holland  types  which  scored  highest  on  the  associated 
DOT  occupations,  taken  in  descending  order,  were  combined  to  generate  the  Holland  code  for 
each  0ES/0*NET  occupation.  The  Holland  codes  for  the  0ES/0*NET  codes  were  reviewed 
along  with  the  codes  for  their  associated  DOT  occupations.  In  some  instances,  this  review 
suggested  that  a DOT  occupation  had  been  linked  to  the  wrong  0ES/0*NET  occupation.  In  these 
cases  0ES/0*NET  Holland  codes  were  recalculated  across  the  corrected  set  of  DOT 
occupations. 

Sager  (1995),  in  summarizing  the  evidence  presented  by  Gottfredson  and  Holland  (1989; 
in  press),  presents  an  example  supporting  the  validity  of  the  DOT  Holland  codes,  as  follows: 

One  example  is  a comparison  between  their  High-point  codes  [Holland  codes]  and 
the  Guide  for  Occupational  Exploration  (GOE;  U.S.  Department  of  Labor,  1979) 
categories.  Employment  Service  occupational  analysts  assigned  the  DOT 

1 Codes  were  derived  for  the  80  occupations  included  in  the  initial  0*NET  occupational  sample.  See  Chapter  2 for 
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occupations  to  12  GOE  categories.  These  categories  represent  1 1 interest 
dimensions  and  1 category  for  occupations  that  require  physical  performance.  The 
U.S.  Department  of  Labor  also  mapped  the  12  GOE  categories  onto  the  six 
Holland  types.  This  allowed  an  examination  of  the  extent  to  which  occupations 
assigned  by  classificatory  function  to  a particular  Holland  category  were  assigned 
by  occupational  analysts  to  the  associated  GOE  category.  The  occupational 
analysts  assigned  76.8  percent  of  the  DOT  occupations  into  the  predicted 
categories,  (p.  11-12) 

This  comparison  was  based  on  Holland  codes  from  the  first  edition  of  the  DHOC  (Gottfredson, 
Holland,  & Ogawa,  1982). 

Finally,  it  is  important  to  note  that  while  Gottfredson  and  Holland  (in  press)  support  the 
validity  of  the  Holland  codes  associated  with  0*NET  occupations,  they  indicate  that  users  should 
consider  these  codes  in  the  context  of  other  available  information,  perhaps  including  direct 
assessments  of  occupations,  “For  large  applications  or  important  individual  decisions...  ”(p.  5-9). 

Results 

The  Holland  codes  that  are  part  of  the  0*NET  prototype  were  not  collected  via 
incumbent/supervisor  or  analyst  questionnaires;  therefore,  the  reliability  and  validity  analyses 
associated  with  the  other  domains  are  not  possible.  However,  the  Holland  codes  shown  in  Figure 
10-2  provide  information  that  allows  profile  comparisons  across  occupations  in  a manner  similar 
to  the  information  presented  for  the  Occupational  Values  Questionnaire  and  for  the  other  domain 
questionnaires  in  other  chapters. 


the  rationale  for  the  selection  of  the  80  occupations. 
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Examination  of  the  Holland  codes  for  the  six  occupations  in  Figure  10-2  provides 
preliminary  evidence  that  the  codes  do  reflect  differences  and  similarities  among  occupations. 

For  example,  Janitors  and  Cleaners,  Except  Maids  and  Housekeeping,  and  Maintenance 
Repairers,  General  Utility  are  the  most  “blue  collar”  of  the  six  occupations,  and  the  first  letter  of 
their  code  is  R for  Realistic.  This  makes  sense  because  occupations  classified  as  primarily  this 
type  "...  tend  to  involve  concrete  and  practical  activity  involving  machines,  tools,  and  materials” 
(Gottfredson  & Holland,  1989,  p.  6). 

However,  the  second  and  third  letters  in  the  codes  for  these  two  occupations  do  not 
match,  distinguishing  these  two  primarily  Realistic  occupations  in  a way  that  makes  sense.  The 
remaining  part  of  the  code  for  Janitors  and  Cleaners,  Expect  Maids  and  Housekeeping  is  E for 
Enterprising  and  then  C for  Conventional.  Enterprising  occupations,  “...  tend  to  involve  working 
with  people  in  a supervisory  or  persuasive  way  to  achieve  some  organizational  goal.” 
(Gottfredson  & Holland,  1989,  p.  6).  Members  of  this  occupation  often  need  to  persuade  people 
to  comply  with  standard  rules  associated  with  the  orderly  functioning  of  a facility  like  walking 
around  barriers  that  indicate  a wet  floor  or  keeping  food  and  drink  out  of  some  parts  of  a 
building.  Conventional  occupations,  “...  tend  to  involve  working  with  things,  numbers,  or 
machines  in  an  orderly  way  to  meet  the  regular  and  predictable  needs  of  an  organization  or  to 
meet  specified  standards”  (Gottfredson  & Holland,  1989,  p.  6).  Janitors  and  Cleaners,  Except 
Maids  and  Housekeeping  generally  have  a relatively  fixed  set  of  tools  and  responsibilities. 

The  second  and  third  letters  of  the  Holland  code  for  Maintenance  Repairers,  General 
Utility  are  I for  Investigative  and  S for  Social.  Occupations  in  the  investigative  category,  tend 
to  involve  analytical  or  intellectual  activity  aimed  at  problem-solving,  trouble-shooting,  or  the 
creation  and  use  of  knowledge”  (Gottfredson  & Holland,  1989,  p.  6).  Repair  involves  finding  and 
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correcting  the  causes  of  equipment  malfunctions.  Social  occupations,  . typically  involve 
working  with  people  in  a helpful  or  facilitative  way”  (Gottffedson  & Holland,  1989,  p.  6).  This 
occupation  often  involves  direct  interaction  with  the  individuals  who  use  the  equipment  being 
repaired. 

The  Holland  codes  for  the  other  occupations  are  also  logically  consistent.  The  first  letter 
of  the  code  for  General  Managers  and  Top  Executives  is  E for  Enterprising.  Supervising  people 
is  part  of  the  definition  of  this  Holland  type.  The  code  for  Computer  Programmers  indicates  that 
it  is  primarily  an  Investigative  occupation.  Debugging  programs  and  the  application  of 
knowledge  of  computer  languages  are  central  parts  of  this  occupation.  Finally,  the  first  letter  of 
codes  for  Registered  Nurses  and  Police  Patrol  officers  is  S for  Social,  correctly  reflecting  the  fact 
that  working  with  and  helping  people  are  important  activities  in  these  occupations. 

Occupational  Values 
Measure 

The  Occupational  Values  Questionnaire  was  included  in  the  0*NET 
incumbent/supervisor  data  collection.  However,  this  questionnaire  was  not  included  in  the 
occupational  analyst  data  collection  effort.2 

The  Occupational  Values  Questionnaire  is  based  on  the  Minnesota  Job  Description 
Questionnaire  (MJDQ;  Borgen,  et  al.,  1968;  Dawis,  1991;  Dawis  & Lofquist,  1984).  The  MJDQ 
describes  occupations  according  to  their  occupational  reinforcer  patterns  (ORPs)  on  21  need- 
remforcers.  Respondents  are  required  to  evaluate  the  relative  extent  to  which  statements 
representing  each  of  these  need-reinforcers  describe  their  occupation.  Each  statement  begins. 
Workers  on  this  job  ...”  For  example,  “Workers  on  this  job  are  busy  all  the  time,”  is  the 
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statement  representing  the  need  reinforcer  “Activity.”  There  are  two  forms  of  the  MJDQ.  In  one 
form  the  statements  are  presented  to  respondents  five  at  a time,  in  a multiple  rank-order  format; 
the  other  form  follows  a paired  comparison  format.  Sager  (1995)  reviews  some  of  the  MJDQ’s 
favorable  reliability  and  validity  evidence  (e.g.,  Borgen,  et  al.,  1968). 

As  described  in  Sager  (1995),  because  of  the  large  number  of  questionnaires  and  items 
included  in  the  0*NET  data  collection  the  judgment  was  made  that  there  was  not  enough  time  or 
space  available  to  use  the  paired  comparison  or  multiple  rank-order  formats.  Based  on  this 
judgment  and  the  results  of  a try-out  study,  an  0*NET  questionnaire  was  developed  that  differs 
from  the  MJDQ  in  three  significant  ways: 

1.  Respondents  are  required  to  rate  each  statement  on  a 5-point  amount  of  agreement 
scale.  The  scale  anchors  are  strongly  disagree,  disagree,  neither  agree  nor  disagree, 
agree,  and  strongly  agree,  respectively. 

2.  The  statements  from  the  MJDQ  for  the  reinforcers  Authority  (Descriptor  #5), 
Company  Policies  (Descriptor  #6),  Co-workers  (Descriptor  #8),  Moral  Values 
(Descriptor  #11),  Social  Status  (Descriptor  #16),  Supervision-Human  Relations 
(Descriptor  #17),  and  Supervision-Technical  (Descriptor  #18)  were  modified  to 
modernize  and  simplify  language. 

3.  The  title  of  the  questionnaire  is  the  Occupational  Values  Questionnaire. 

The  MJDQ’s  21  reinforcers  and  the  statements  used  in  the  Occupational  Values  Questionnaire 
are  presented  in  Figure  10-3.  The  complete  Occupational  Values  Questionnaire  is  presented  in 
the  Development  of  Prototype  Occupational  Information  Network  (Q*NET>  Content  Model 
report  (Peterson,  Mumford,  Borman,  Jeanneret,  & Fleishman,  1995). 


2 Occupational  analysts  completed  ratings  using  a modified  version  of  the  Occupational  Values  Questionnaire  as  part 
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Taxonomy 

The  21  reinforcers  represent  the  first  level  of  0*NET’s  occupational  values  taxonomy. 
The  titles  and  definitions  of  the  dimensions  representing  the  hypothesized  second  level  of  the 
taxonomy  and  the  reinforcers  (i.'e.,  descriptors)  associated  with  each  of  these  higher-level 
dimensions  are  presented  in  Figure  10-4.  This  structure  came  from  Dawis’s  (1991)  review  of 
studies  that  factor  analyzed  responses  to  the  Minnesota  Importance  Questionnaire  (MIQ;  Dawis 
& Lofquist,  1984).  The  MIQ  is  designed  to  measure  people,  not  occupations.  As  indicated 
previously,  it  consists  of  the  same  need-reinforcer  statements  as  the  MJDQ;  however, 
respondents  are  required  to  rank  order  sets  of  the  five  statements  in  terms  of  their  relative 
importance  in  an  “ideal”  job.  The  results  of  these  factor  analyses  represent  the  higher  level  of  the 
taxonomy  for  the  importance  of  reinforcers  in  people’s  ideal  occupations.  Therefore,  it  is  a 
working  hypothesis  that  this  structure  applies  equally  well  as  the  higher  level  of  the  taxonomy  for 
ratings  of  the  presence  of  reinforcers  in  actual  occupations. 

Sample 

Of  the  80  occupations  targeted  in  the  0*NET  prototype  incumbent/supervisor  sample, 
only  36  yielded  four  or  more  completed  Occupational  Values  Questionnaire.  Only  these 
respondents  in  those  36  occupations  were  included  in  the  analyses,  for  a total  of  682  respondents. 
For  each  occupation.  Figure  10-5  lists  the  0*NET  occupation  code,  occupation  title,  and  the 
number  of  respondents  per  occupation.  The  36  occupations  represent  a broad  range  of 
occupations  in  terms  of  level  and  industry. 


of  another  project,  but  those  results  are  not  discussed  here. 
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Results 

Descriptive  Statistics 

Table  10-1  presents  some  descriptive  statistics  for  each  of  the  items  on  the  Occupational 
Values  Questionnaire.  Consistent  with  the  description  above,  each  item  is  a statement  that  carries 
the  descriptor  label  of  its  associated  need-reinforcer,  and  responses  to  the  items  were  ratings  on  a 
. 5-point  agreement  scale.  The  second  column  of  Table  10-1  shows  the  mean  of  the  mean  ratings 
for  each  occupation  on  each  descriptor.  These  means  show  that  overall  the  ratings  are 
concentrated  in  the  top  half  of  the  scale.  However,  the  standard  deviations  of  the  mean  ratings 
across  occupations  in  Table  10-1  show  that  there  is  some  variation  across  occupations  in  terms  of 
agreement  about  the  presence  of  reinforcers.  The  fourth  and  fifth  columns  of  Table  10-1  show  the 
standard  error  of  estimate  and  reliability  of  the  mean  occupation  rating  for  each  descriptor. 

Reliability 

The  standard  deviations  in  Table  10-1  suggest  that  occupations  do  vary  on  the  reinforcers 
and  that  they  might  be  useful  for  discriminating  among  occupations.  To  what  extent  does  this 
variation  represent  reliable  differences  among  occupations?  The  results  presented  in  Tables  1 
through  7 address  the  reliability  of  the  mean  occupation  ratings  on  the  Occupational  Values 
Questionnaire. 

The  standard  error  of  measurement  (SEMI  and  reliability  estimates  in  Table  10-1  provide 
a mixed  picture.  The  SEMs  are  generally  not  small  relative  to  the  standard  deviations,  indicating 
that  a substantial  part  of  the  variation  in  mean  ratings  across  occupations  is  due  to  unreliability. 
The  reliabilities,  based  on  a harmonic  mean  of  k = 9.69  raters  per  occupation,  are  also  not 
generally  high;  the  reliabilities  for  eight  descriptors  are  below  = .50.  However,  the  reliabilities 
for  five  of  the  descriptors  are  above  & = .70.  The  mean  reliability  is  .53,  the  median  is  .55.  It  is 
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relevant  to  note  that  reliabilities  depend  on  the  number  of  raters  per  occupation.  Table  10-2 
provides  estimates  of  what  the  reliabilities  for  the  mean  occupations  ratings  would  be  if  they 
were  based  on  1 and  30  raters  per  occupation.  The  single  rater  reliabilities  are  all  low;  the  highest 
is  rj  = .28.  Predictably,  the  30-rater  reliabilities  are  substantially  higher  than  the  k rater 
reliabilities.  For  14  of  the  descriptors  they  are  £30  = .75  or  greater;  however,  some  are  low.  For 
example,  the  30-rater  reliability  for  Moral  Values  (Descriptor  #1 1)  is  still  only  £30  = .26.  These 
results  indicate  that  reliable  occupation  mean  ratings  on  individual  Occupational  Values 
Questionnaire  descriptors  depend  on  at  least  30  raters  per  occupation. 

Analyses  of  Variance 

Tables  10-4  through  10-7  show  another  way  to  address  the  reliability  issue.  Analysis  of 
variance  was  used  to  separately  calculate  the  effects  of  occupations,  raters  within  occupations, 
descriptors,  descriptor  by  occupation  interactions,  and  descriptor  by  raters  within  occupations 
interactions  on  the  variation  in  ratings.  Table  10-4  shows  that  ratings  differ  significantly  across 
occupations.  This  result  supports  the  hypothesis  that  the  overall  presence  of  reinforcers  varies 
from  one  occupation  to  the  next.  Table  10-4  also  shows  that  ratings  differ  significantly  across 
descriptors.  This  result  supports  the  hypothesis  that  across  all  occupations,  some  reinforcers  are 
more  likely  to  be  present  than  others.  Finally,  Table  10-4  indicates  a significant  interaction 
between  the  effect  of  descriptors  on  ratings  and  the  effect  of  occupations  on  ratings.  This  result 
supports  the  hypothesis  that  generally  the  presence  of  particular  reinforcers  varies  across 
occupations,  suggesting  that  the  occupational  values  instrument  can  discriminate  among 
occupations. 

Table  10-5  shows  the  interrater  agreement  coefficients  based  on  the  results  presented  in 
Table  10-4  for  k raters  per  occupation,  one  rater  per  occupation,  and  30  raters  per  occupation. 
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The  overall  interrater  agreement  for  the  questionnaire,  based  on  a harmonic  mean  of  k = 9.69 
raters  per  occupation  is  not  high  (i*  = .60),  and  the  interrater  agreement  for  one  rater  per 
occupation  is  low  (rj  = .13),  however,  the  overall  interrater  agree  for  30  raters  per  occupation  is 
fairly  strong  (130  = .82).  This  result  suggests  that,  with  30  raters  per  occupation,  the  pattern  of 
scores  on  the  descriptors  in  the  Occupational  Values  Questionnaire  could  reliably  discriminate 
among  pccupations. 

As  shown  in  Figure  10-4,  a higher  level  was  hypothesized  for  the  taxonomy  of  reinforcers 
in  the  Occupational  Values  Questionnaire.  The  mean  first-order  ratings  for  each  occupation  were 
aggregated  to  create  scores  for  each  occupation  on  the  six  dimensions  described  in  Figure  10-4. 
The  analysis  of  variance  and  overall  interrater  agreement  results  for  the  six  aggregate  scores  are 
presented  in  Tables  10-6  and  10-7.  Table  10-6  shows  that  aggregate  scores  vary  significantly 
across  occupations,  aggregates,  and  that  the  interaction  between  occupations  and  aggregates  is 
significant.  However,  a comparison  between  the  results  in  Tables  10-5  and  10-7  indicates  that  the 
level  of  interrater  agreement  is  virtually  the  same  for  the  6 aggregates  as  it  is  for  21  reinforcer 
scores. 

Descriptor  and  Scale  Relationships 

Table  10-9  presents  the  correlations  among  the  ratings  at  the  occupation  level,  and  Table 
10-10  presents  these  correlations  at  the  individual  level  based  on  four  randomly  selected  raters 
per  occupation.  Examination  of  correlations  among  the  ratings  at  the  occupational  level  is 
emphasized  here  for  two  reasons:  (1)  the  primary  concern  here  is  the  study  of  occupations  and  (2) 
the  reliabilities  of  the  ratings  underlying  the  correlations  among  ratings  at  the  individual  level  are 
so  low  that  attempts  to  discern  patterns  are  not  likely  to  be  fruitful. 
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Examination  of  the  correlations  in  Table  10-9  indicates  that  the  relationships  among  the 
reinforcer  ratings  are  generally  positive  with  two  notable  exceptions:  (1)  Independence 
(Descriptor  #10)  is  negatively  correlated  with  almost  all  of  the  other  ratings,  and  (2)  there  is  a 
relatively  high  negative  correlation  between  Social  Service  (Descriptor  #15)  and  Compensation 
(Descriptor  #7)  (r  = -.48).  Examination  of  the  correlations  relative  to  the  hypothesized  higher 
le^vel  structure  shown  in  Figure  10-4  shows  that  the  correlations  are  not  particularly  consistent 
with  this  structure  with  the  exception  of  the  first  hypothesized  aggregate  “Achievement;”  its  two 
constituent  reinforcers  (i.e..  Ability  Utilization  [Descriptor  #1]  and  Achievement  [Descriptor  #2]) 
show  a relatively  high  correlation  (r  = .77).  The  highest  mean  correlation  for  any  of  the  other 

aggregates  is  .45  for  Safety.  Comfort  is  a mixture  of  positive  and  negative  correlations,  with  a 
mean  of  .04. 

Factor  Structure 

Table  10-11  presents  the  pattern  matrix  from  a principal  components  analysis  of  the 
occupation  level  correlation  matrix  in  Table  10-9,  after  an  orthogonal  varimax  rotation. 
Examination  of  eigenvalues  and  interpretability  of  the  solutions  favors  a five  factor  solution  that 
accounts  for  71%  of  the  variance  in  mean  ratings.  The  communalities  indicate  that  five  factors 
generally  account  for  a substantial  amount  of  the  variance  in  mean  occupation  ratings  with  the 
exception  of  Working  Conditions  (Descriptor  #20)  whose  communality  is  only  .39. 

Factor  1 is  labeled  individual  accomplishment  and  accounts  for  25%  of  the  variance  in 
mean  occupation  ratings.  This  factor  subsumes  the  hypothesized  aggregates  Achievement  and 
Autonomy,  shown  in  Figure  10-4,  because  the  highest  loadings  for  the  reinf orcers  Ability 
Utilization  (Descriptor  #1)  (.87),  Achievement  (Descriptor  #2)  (.87),  Creativity  (Descriptor  #9) 
(.88),  Responsibility  (Descriptor  #13)  (.77),  and  Autonomy  (Descriptor  #21)  (.71)  are  on  this 
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factor.  The  highest  loadings  for  Authority  (Descriptor  #5)  (.79)  and  Variety  (Descriptor  #19) 

(.78)  are  also  on  Factor  1. 

Factor  2 accounts  for  12%  of  the  variance  in  mean  occupation  ratings  and  is  labeled 
structure.  This  factor  shares  some  of  the  reinforcers  associated  with  the  hypothesized  aggregates 
comfort  and  safety.  The  highest  loadings  for  four  reinforcers  are  on  this  factor  (1)  Activity 
(Descriptor  #3)  (.61),  (2)  Independence  (Descriptor  #10)  (-.78),  (3)  Supervision  - Human 
Relations  (Descriptor  #17)  (.71),  and  (4)  Supervision  - Technical  (Descriptor  #18)  (.60). 
Company  Policies  (Descriptor  #6)  (.46)  also  shows  a relatively  high  loading  on  this  factor. 

Factor  3 is  labeled  social  comfort  and  accounts  for  12%  of  the  variance  in  mean 
occupation  ratings.  This  factor  shares  some  of  the  reinforcers  associated  with  the  hypothesized 
aggregates  Altruism  and  Safety.  However,  the  reinforcers  with  high  loadings  on  Factor  3 together 
appear  to  describe  the  extent  to  which  occupations  are  socially  comfortable.  Reinforcers  with 
high  loadings  on  this  factor  include  Company  Policies  (Descriptor  #6)  (.48),  Co-workers 
(Descriptor  #8)  (.76),  Moral  Values  (Descriptor  #1 1)  (.66),  Recognition  (Descriptor  #12)  (.62), 
Supervision  - Human  Relations  (Descriptor  #17)  (.41),  and  Working  Conditions  (Descriptor  #20) 
(.61). 

Factor  4 accounts  for  12%  of  the  variance  in  mean  occupation  ratings  and  is  labeled 
career  advancement.  This  factor  is  not  strongly  related  to  any  of  the  hypothesized  aggregates  in 
terms  of  shared  reinforcers.  The  reinforcers  Advancement  (Descriptor  #4)  (.73)  and 
Compensation  (Descriptor  #7)  (.83)  show  the  highest  loadings  on  this  factor.  Company  Policies 
(Descriptor  #6)  (.40)  and  Recognition  (Descriptor  #12)  (.41)  also  show  high  loadings  on  this 
factor.  Social  Service  (Descriptor  #15)  (-.69)  loads  negatively  on  Factor  4 suggesting  that,  in  this 
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sample,  occupations  that  pay  well  and  provide  opportunities  for  Advancement  (Descriptor  #4)  do 
not  emphasize  Social  Service  (Descriptor  #15)  and  vice  versa. 

Factor  5,  labeled  stability,  accounts  for  1 1%  of  the  variance  in  mean  occupation  ratings. 
This  factor  is  also  not  strongly  related  to  any  of  the  hypothesized  aggregates.  Loadings  for  two 
reinforcers  are  highest  on  this  factor:  Security  (Descriptor  #14)  (.77)  and  Social  Status 
(Descriptor  #16)  (.77).  Social  Service  (Descriptor  #15)  (.45)  also  has  a high  loading  on  Factor  5. 
This  factor  seems  to  capture  the  degree  of  stability  in  the  organization  and  community  that  the 
occupation  offers. 

Overall  the  results  of  the  principal  components  analysis  are  interpretable.  They  suggest 
that  meaningful  constructs  are  being  assessed  by  the  Occupational  Values  Questionnaire  and  that 
the  mean  ratings  on  the  reinforcers  can  discriminate  among  occupations  in  meaningful  ways. 
However,  the  empirically  discovered  structure  of  the  occupation  mean  ratings  is  different  from 
the  hypothesized  structure.  There  are  three  salient,  possible  reasons  for  this  difference.  First,  the 
hypothesized  structure  is  based  on  factor  analyses  of  the  MIQ,  an  instrument  designed  to  measure 
the  importance  of  reinforcers  in  people’s  ideal  occupations,  not  the  presence  of  reinforcers  in 
actual  occupations.  Second,  as  previously  stated  in  this  chapter  and  discussed  in  detail  in  Sager 
(1995),  some  of  the  statements  (i.e.,  items)  in  the  Occupational  Values  Questionnaire  are 
modifications  of  those  in  the  MJDQ;  changes  in  wordings  may  have  affected  the  relationships 
among  some  of  the  reinforcers.  Finally,  the  sample  of  682  respondents  in  36  occupations  may  be 
somewhat  idiosyncratic. 


Another  method  of  examining  the  factor  structure  of  the  Occupational  Values 
Questionnaire  is  confirmatory  factor  analysis.  Two  competing  models  were  hypothesized  to 
underlie  the  occupation  level  correlations  among  the  reinforcers  in  Table  10-9.  Windows 
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LISREL  8.03  was  used  to  test  these  models  (Joreskog  & Sorbom,  1993).  The  first  model  is 
derived  from  the  hypothesized  higher  level  of  the  Occupational  Values  Questionnaire  that  is 
summarized  in  Figure  10-4.  As  indicated  previously,  six  factors  were  hypothesized  based  on 
factor  analyses  of  the  MIQ.  After  1,000  iterations  LISREL  was  not  able  to  produce  final 
maximum-likelihood  parameter  estimates  for  this  model.  The  second  model  is  derived  from  the 
results  of  the  exploratory'  principal  components  analysis  (see  Table  10-1 1).  LISREL  was  able  to 
generate  maximum-likelihood  estimates  of  the  parameters  for  this  model;  however,  the  fit  was 
poor  (x2(184,  N = 36)  = 334.06,  p < .001).  The  pattern  matrices  and  factor  correlation  matrices 
for  these  models  are  not  presented  here  because  they  fit  the  data  so  poorly. 

However,  it  is  important  to  note  that  the  principal  components  analysis  and  the  two 
confirmatory  factor  analyses  do  not  represent  the  best  possible  examination  of  the  factor 
structures  underlying  the  relationships  among  the  reinforcers.  First,  the  occupation  mean  ratings 
are  not  uniformly  reliable;  some  are  based  on  as  few  as  four  raters  per  occupation.  The  results  of 
the  reliability  analysis  make  it  clear  that  more  than  four  raters  per  occupation  are  needed  to 
obtain  reliable  reinforcer  occupation  means.  Second,  only  36  occupations  underlie  the 
correlations  in  the  matrix  that  was  analyzed;  this  means  that  all  three  analyses  were  estimating 
more  parameters  than  there  were  cases.  In  consideration  of  these  two  points,  the  results  of  these 
analyses  should  be  considered  preliminary. 

Occupation  Differences 

Table  10-12  offers  a demonstration  of  the  extent  to  which  ratings  on  the  Occupational 
Values  Questionnaire  can  capture  the  similarities  and  differences  among  occupations  according 
to  its  21  reinforcers.  This  table  presents  the  mean  ratings  and  standard  deviations  for  six  distinct 
occupations:  (1)  General  Managers  and  Top  Executives,  (2)  Computer  Programmers,  (3) 
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Registered  Nurses,  (4)  Police  Patrol  Officers,  (5)  Janitors  and  Cleaners,  Except  Maids  and 
Housekeeping,  and  (6)  Maintenance  Repairers,  General  Utility. 

Table  10-12  shows  that  mean  ratings  of  a number  of  the  reinforcers  vary  little  across  the 
six  occupations.  For  example,  the  range  of  mean  ratings  for  four  reinforcers  on  the  5-point 
agreement  scale  is  .50  or  less  across  these  occupations:  Ability  Utilization  (Descriptor  #1),  Co- 
workers (Descriptor  #8),  Supervision  - Human  Relations  (Descriptor  #17),  and  Autonomy 
(Descriptor  #21).  This  result  suggests  that  these  six  occupations  are  generally  similar  with  regard 
to  the  presence  of  these  reinforcers  and  that  they  do  not  discriminate  well  among  these 
occupations. 

However,  some  of  the  reinforcers  do  vary  across  these  six  occupations.  For  example,  the 
range  of  mean  ratings  of  Social  Service  (Descriptor  #15)  across  the  six  occupations  is  1.83.  The 
mean  Social  Service  (Descriptor  #15)  rating  is  near  M = 4.00  for  five  of  the  occupations; 
however,  it  is  relatively  low  for  Computer  Programmers  (M  = 2.86,  SD  = .90).  Likewise,  the 
range  of  mean  ratings  on  Social  Status  (Descriptor  #16)  is  1.39.  The  mean  ratings  on  this 
reinforcer  were  lower  for  Computer  Programmers  (M  = 2.86,  SD  = .90)  and  Janitors  and 
Cleaners,  Except  Maids  and  Housekeeping  (M  = 2.93,  SD  = 1.07)  than  for  Police  Patrol  Officers 
(M  = 4.25,  SD  = .68),  General  Mangers  and  Top  Executives  (M  = 3.93,  SD  = .70).  Finally,  the 
range  of  mean  ratings  on  Activity  (Descriptor  #3)  is  1.19.  General  Managers  and  Top  Executives 
showed  the  highest  mean  rating  on  Activity  (Descriptor  #3)  (M  = 4.44,  SD  = .70),  while  Police 
Patrol  Officers  showed  the  lowest  (M  = 3.25,  SD  = .90).  On  Advancement  (Descriptor  #4),  there 
is  a difference  of  1.27  between  the  mean  ratings  for  Computer  Programmers  (higher)  1 Janitors 
and  Cleaners.  These  example  comparisons  of  mean  ratings  on  occupation  reinforcers  are 
consistent  with  expected  similarities  and  differences  among  the  six  occupations  included  in 
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Table  10-12.  And,  the  reinforcers  discussed  in  this  paragraph  are  among  those  identified  in  the 
discriminant  analyses  as  differentiating  among  occupations. 

Discriminant  Analysis 

Discriminant  analysis  is  another  method  to  ask  the  question,  “Does  the  Occupational 
Values  Questionnaire  differentiate  among  occupations?”  Table  10-13  presents  the  results  of  a 
discriminant  analysis  based  on  k raters  from  each  of  the  36  occupations  included  in  the  analyses 
for  this  chapter.  Table  10-13  shows  (1)  the  correlation  between  each  reinforcer  and  the  first  six 
rotated  discriminant  functions,  (2)  the  sum  of  the  squared  rotated  correlations  between  the 
discriminating  variables  (i.e.,  reinforcers)  across  six  discriminant  functions  (XF2),  (3)  the 
proportion  of  variance  in  the  ratings  of  each  reinforcer  accounted  for  by  occupations  (t|2),  (4)  the 
canonical  correlation  for  each  function  (Rc),  (5)  the  percent  of  variance  accounted  for  by  each 
function,  and  (6)  the  eigenvalue  for  each  function. 

The  first  six  functions  account  for  71%  of  the  between  occupations  variance  in  ratings. 
All  21  classification  functions  correctly  reassigned  35%  of  the  raters  into  the  occupations  from 
which  they  were  drawn.  This  correct  assignment  rate  is  substantially  above  chance,  which  would 
be  about  3%  for  36  occupations,  if  all  the  occupations  were  of  equal  size. 

The  correlations  between  the  reinforcers  and  the  first  six  functions,  IF2  values,  and  q2 
coefficients  show  that  these  discriminant  functions  are  dominated  by  six  reinforcers.  The 
correlation  between  Function  1 and  Activity  (Descriptor  #3)  is  r = .98.  The  correlation  between 
Function  2 and  Social  Status  (Descriptor  #16)  is  r = .93.  The  correlation  between  Function  3 and 
Authority  (Descriptor  #5)  is  r = .96.  The  correlation  between  Function  4 and  Advancement 
(Descriptor  #4)  is  r = .94.  The  correlation  between  Function  5 and  Social  Service  (Descriptor 
#15)  is  r = .98.  And  finally,  the  correlation  between  Function  6 and  Security  (Descriptor  #14)  is  r 
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= .97.  None  of  the  other  correlations  between  reinforcers  and  functions  are  greater  than  r = .15. 
These  results  suggest  that  these  are  the  six  best  reinforcers  in  terms  of  differentiating  among  the 
occupations  in  this  sample. 

Additional  Validity  Evidence 

As  noted  previously,  the  Occupational  Values  Questionnaire  is  a modification  of  the 
MJDQ.  There  are  two  important  characteristics  of  the  MJDQ  to  note:  (1)  occupations  are 
primarily  compared  by  occupational  reinforcer  profile,  not  by  individual  reinforcers  (e.g.,  Borgen 
et  al.,  1968;  Dawis  & Lofquist,  1984)  and  (2)  researchers  suggest  that  stable  results  depend  on  a 
minimum  of  20  raters  per  occupation  (e.g.,  Borgen  et  al.,  1968).  As  indicated  previously,  the 
results  presented  in  this  chapter  are  based  on  N = 36  occupations,  with  a median  of  only  12  raters 
per  occupation.  This  small  sample  does  not  provide  strong  evidence  in  support  of  the  reliability 
of  the  Occupational  Values  Questionnaire.  However,  Table  10-5  shows  that  the  estimated  overall 
reliability  of  the  questionnaire  would  be  £30  = .82  if  there  were  30  raters  per  occupation.  This 
estimated  reliability  considered  along  with  these  two  characteristics  of  the  MJDQ  suggests  an 
alternative  method  of  examining  the  reliability  and  validity  of  the  Occupational  Values 
Questionnaire. 

Table  10-17  summarizes  the  results  of  an  alternative  method  of  examining  the  reliability 
and  validity  of  the  Occupational  Values  Questionnaire  that  compares  mean  group  profiles  within 
and  across  occupations.  Three  of  the  36  occupations  included  in  the  sample  were  rated  by  close 
to  or  more  than  60  raters.  The  occupations  are  (1)  First  Line  Supervisors,  Clerical/Administrative 
(n  = 59),  (2)  Secretaries,  Except  Legal  and  Medical  (n  = 67),  and  (3)  General  Office  Clerks  (n  = 
92).  Within  each  occupation  respondents  were  randomly  assigned  to  two  groups  of  30  each, 
except  that  the  first  group  of  First  Line  Supervisors,  Clerical/ Administrative  contained  only  29 


1061 


Occupational  Interests  and  Values  10-21 


raters.  Profiles  were  calculated  for  each  of  the  six  groups  (i.e.,  two  groups  for  each  of  three 
occupations).  Each  profile  consists  of  the  mean  ratings  for  that  group  on  each  of  the  21 
reinforcers.  A matrix  containing  the  correlations  among  the  six  group  profiles  was  then 
calculated. 

To  achieve  more  stable  estimates  of  these  correlations,  the  process  of  assigning  raters 
randomly  to  one  of  two  groups  within  each  occupation  was  performed  10  times  resulting  in  10 
different  randomly  assigned  sets  of  6 groups.  A correlation  matrix  was  calculated  for  each  set  of 
six  group  profiles.  Table  10-17  presents  the  median  correlations  across  the  10  correlation 
matrices. 

The  within  occupation  median  correlations  between  group  profiles  are  £30  = .86,  £30  = .87, 
and  £30  = .91,  respectively.  These  correlations  are  estimates  of  the  overall  reliability  of  the 
Occupational  Values  Questionnaire  for  these  occupations  when  there  are  30  raters  per 
occupation.  These  reliabilities  are  good  and  similar  to  the  projected  reliability  presented  in  Table 
10-5  (£30  = .82). 

The  question  remains,  however,  “Can  profiles  on  this  questionnaire  capture  similarities 
and  differences  between  occupations?”  The  remaining  off-diagonal  median  correlations  in  Table 
10-17  address  this  question.  They  are  correlations  between  group  profiles  across  occupations. 
There  are  four  such  correlations  for  each  pair  of  occupations.  For  example,  the  between  group 
and  across  occupation  median  profile  correlations  for  Occupations  1 and  2 ( i.e.,  First  T.inp 
Supervisors,  Clerical/Administrative  and  Secretaries  Except  Legal  and  Medical,  respectively)  are 
I = -69,  £ = .64,  £ = .74,  and  £ = .70.  These  values  are  lower  than  the  within  occupation  madjan 
correlations  for  these  two  occupations  (£30  = .86  and  £30  = .87,  respectively).  This  result  is 
evidence  that  this  questionnaire  recognizes  differences  between  these  two  occupations,  even 
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though  they  tend  to  occur  in  the  same  environment  and  involve  many  of  the  same  tasks.  The 
between  occupation  median  correlations  comparing  Occupation  1 profiles  to  Occupation  3 (i.e.. 
General  Office  Clerks)  profiles  are  also  somewhat  lower  than  the  relevant  within  occupation 
median  correlations,  suggesting  that  this  questionnaire  can  differentiate  between  these  two 
occupations. 

However,  the  median  correlations  comparing  Occupation  2 profiles  to  Occupation  3 
profiles  are  high  relative  to  their  respective  within  occupation  median  correlations.  This  result 
suggests  that  this  questionnaire  does  not  differentiate  Occupation  2 from  Occupation  3 in  terms 
of  reinforcers.  This  is  not  surprising  given  the  similarities  in  these  occupations;  Occupation  2 is 
regular  Secretaries  and  Occupation  3 is  Office  Clerks. 

Taken  as  a whole,  the  results  presented  in  Table  10-17  provide  evidence  that  the 
Occupational  Values  Questionnaire  can  reliably  describe  occupations  in  terms  of  reinforcers  if  it 
is  supported  by  30  raters  per  occupation.  It  is  also  interesting  that  the  questionnaire  was  able  to 
partially  differentiate  among  three  occupations  that  take  place  in  same  environment  and  involve 
very  similar  tasks;  all  three  are  clerical  office  occupations.  Smaller  between  occupation  profile 
correlations  would  be  likely  if  less  similar  occupations  were  considered. 

As  described  in  Sager  (1995),  Borgen  et  al.  (1968)  used  this  method  to  examine 
the  reliability  and  validity  of  the  ranked  version  of  the  MJDQ. 

Their  study  included  8 1 occupations.  For  each  occupation,  two  ORPs  were 
created,  each  based  on  half  of  the  respondents  completing  the  questionnaire  for 
that  particular  occupation.  The  within  occupation  correlations  between  groups 
ranged  from  r = .78  to  r.=  .98  with  a median  of  r = .91  (p.  1 1-10). 
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The  median  correlation  between  profiles  from  different  occupations  was  r = .55  (Borgen 
et  al.,  1968).  This  result  provided  evidence  that  the  MJDQ  could  reliably  differentiate  among 
occupations  because  the  correlations  between  profiles  were  on  average  higher  within  occupations 
than  between  occupations.  It  also  provides  additional  promise  for  the  Occupational  Values 
Questionnaire  in  future  samples  that  will  include  a wider  variety  of  occupations  with  more  raters 
per  occupation. 

Conclusions 

The  primary  motivation  for  describing  occupations  in  terms  of  interests  and  occupational 
values  is  to  facilitate  the  match  between  the  interests  and  values  and  occupations.  That  is, 
occupations  vary  according  to  the  activities  they  involve  and  the  potential  reinforcers  that  they 
offer  incumbents.  Two  taxonomies  and  their  associated  indicators  were  discussed  in  this  chapter: 
(1)  Holland  codes  that  denote  the  three  Holland  types  most  closely  associated  with  each 
occupation  and  (2)  mean  occupation  ratings  on  the  21  need-reinforcers  presented  in  the 
Occupational  Values  Questionnaire.  The  goal  was  to  assess  the  feasibility  of  including  these 
constructs  and  their  indicators  in  O*  NET’s  content  model.  In  this  context  there  are  three  parts  to 
the  feasibility  question: 

1 . Do  research  and  theory  support  the  selected  occupational  interest  and  values 
taxonomies? 

2.  Are  the  procedures  for  assessing  occupations  according  to  the  selected  interest  and 
values  taxonomies  practically  useable  in  the  context  of  0*NET? 

3.  Are  the  assessments  of  the  occupations  according  to  the  selected  interest  and  values 
taxonomies  reliable  and  valid? 
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The  rest  of  this  chapter  summarizes  the  feasibility  evidence  and  discusses  its  implications  for 
potential  revisions  to  the  content  model  and  measures. 

Interests 

As  described  in  Sager  (1995),  the  six  Holland  types  are  a significant  part  of  the  vocational 
and  career  counseling  literature  and  there  is  evidence  supporting  the  reliability  and  validity  of 
various  assessments  of  occupations  according  to  the  Holland  types.  Sager  (1995)  also  indicates 
that  a good  deal  of  this  research  depends  on  large  numbers  of  individuals  completing  long 
instruments  for  each  occupation  in  question.  The  alternative  procedure  that  Gottfredson  and 
Holland  (in  press)  used  for  developing  Holland  codes  for  0ES/0*NET  occupations  is  relatively 
efficient  and  has  already  generated  Holland  codes  for  every  occupation  in  this  structure. 

Sager  (1995)  summarizes  an  example  of  the  evidence  supporting  the  validity  of  the 
Holland  codes  for  DOT  occupations  (Holland  & Gottfredson  1989;  in  press).  This  chapter 
examines  the  validity  of  the  Holland  codes  for  0*NET  occupations  by  reviewing  the  codes  for 
six  occupations  that  vary  across  industry  and  level  (see  Figure  10-2).  The  codes  describe  and 
differentiate  among  the  occupations  in  a manner  that  is  logically  consistent  with  definitions  of 
each  of  the  Holland  types  and  with  what  is  known  about  these  occupations. 

Recommendations 

The  evidence  supporting  the  validity  of  the  Holland  codes  developed  by  Gottfredson  and 
Holland  (in  press)  is  positive,  yet  preliminary.  There  is  a fair  amount  of  evidence  supporting  the 
validity  of  the  these  codes  for  DOT  occupations,  but  more  evidence  is  needed  to  support  their 
validity  for  the  0*NET  occupations.  Analyses  comparing  these  Holland  codes  with  other  parts  of 
the  0*NET  content  model  could  substantially  increase  confidence  in  the  validity  of  the  Holland 
codes.  However,  given  the  positive  preliminary  evidence,  the  substantial  literature  behind  the 
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Holland  types,  and  the  wide  use  of  Holland  taxonomy  in  vocational  and  career  counseling  it  is 
recommended  that  the  Holland  codes  remain  part  of  0*NET’s  content  model. 

Occupational  Values 

Sager  (1995)  summarizes  the  literature  supporting  the  need  to  describe  occupations  in 
terms  of  the  values  (i.e.,  the  importance  of  particular  work  activities  and  other  characteristics  of 
the  work  environments;  Dawis,  1991).  Sager  also  reviews  the  MJDQ,  an  instrument  designed  to 
describe  occupations  according  to  the  relative  presence  of  21  need-reinforcers  (i.e.,  work  values). 
Dawis  (1991)  provides  evidence  that  these  21  reinforcers  address  the  work  values  assessed  by 
other  instruments.  Borgen  et  al.  (1968),  Dawis  and  Lofquist  (1984),  and  (Dawis,  1991)  describe 
and  summarize  evidence  supporting  the  reliability  and  validity  of  the  MJDQ.  Finally,  Weiss, 
Dawis,  England,  and  Lofquist  (1964)  and  Gay,  Weiss,  Hendel,  Dawis,  and  Lofquist  (1971) 
provide  evidence  that  the  statements  presented  in  the  MJDQ  to  assess  the  reinforcers  are  easy  to 
read  and  understand.  Clearly,  then,  the  Occupational  Values  Questionnaire  is  based  on  sound 
theoretical  ground  and  a very  practical  and  useable  occupation  assessment  instrument  (i.e.,  the 
MJDQ). 

There  are,  however,  some  concerns  about  the  Occupational  Values  Questionnaire  and  the 
nature  of  the  sample  of  occupations  and  respondents  included  in  the  0*NET  prototype  data 
collection  effort.  First,  as  mentioned  previously,  the  Occupational  Values  Questionnaire  is  based 
on  the  MJDQ  with  two  substantial  modifications: 

L The  MJDQ  requires  forced  choices  between  or  among  statements  that  represent  each 
of  the  21  reinforcers,  while  the  Occupational  Values  Questionnaire  requires  that  each 
statement  be  individually  rated. 
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2.  In  the  Occupational  Values  Questionnaire  the  wording  of  some  of  the  21  MJDQ 
statements  was  modified. 

These  modifications  have  implications  for  consideration  of  the  results  of  the  analyses.  As 
discussed  in  Sager  (1995),  if  respondents  are  not  forced  to  make  choices  about  reinforcers 
relative  to  each  other  they  may  display  (a)  a positive  bias  and  rate  every  reinforcer  as  present  in 
their  occupation  or  (b)  a job  satisfaction  bias  and  rate  every  reinforcer  according  to  their  overall 
level  of  job  satisfaction.  The  concern  was  that  this  would  result  in  mean  occupation  ratings  that 
were  uniformly  high  and/or  showed  little  variation  across  occupations  and  reinforcers.  Such  a 
result  could  reduce  the  effectiveness  of  the  ratings  on  the  Occupational  Values  Questionnaire  in 
terms  of  its  ability  to  differentiate  among  occupations.  Additionally,  there  is  some  concern  that 
modifications  to  the  wording  of  the  reinforcer  statements  may  affect  the  construct  that  each 
assesses  and  thus  change  the  structure  of  the  taxonomy. 

There  are  two  concerns  about  the  data  available  for  these  analyses.  The  first  is  that,  while 
the  targeted  sample  in  the  initial  0*NET  data  collection  included  80  of  the  more  populous 
occupations,  unfortunately,  for  only  36  of  the  occupations  were  the  minimum  of  four 
Occupational  Values  Questionnaires  returned.  These  occupations  represent  a broad  range  of 
occupations  in  terms  of  level  and  industry;  however,  it  is  difficult  to  infer  that  this  small  non- 
random  sample  of  occupations  represents  the  full  variety  of  occupations  in  terms  of  interests  or 
occupational  values.  Another  difficulty  is  the  small  number  of  respondents  per  occupation.  The 
literature  indicates  that  the  MJDQ  needs  20  raters  per  occupations  to  generate  reliable 
descriptions  of  occupations.  The  median  number  of  respondents  per  occupation  across  the  36 
occupations  was  only  12,  and  for  some  occupations,  as  few  as  four  incumbents  responded. 


Occupational  Interests  and  Values  10-27 


Unfortunately,  these  data  do  not  allow  for  an  optimal  test  of  the  reliability  and  validity  of  the 
Occupational  Values  Questionnaire. 

It  is  in  the  context  of  these  strengths  and  weaknesses  that  the  evidence  regarding  the 
reliability  and  the  validity  of  the  Occupational  Values  Questionnaire  should  be  considered.  The 
tables  in  this  chapter  present  evidence  regarding  the  extent  to  which  the  occupation  mean  ratings 
on  the  questionnaire  can  reliably  differentiate  among  occupations.  Tables  1 through  5 present 
reliability  evidence  in  terms  of  the  extent  to  which  there  is  more  variation  in  respondent  ratings 
across  occupations  compared  to  within  ratings.  This  method  of  considering  reliability,  referred  to 
as  interrater  agreement,  assumes  that  occupations  vary  on  the  constructs  being  assessed. 
Therefore,  these  indices  of  interrater  agreement  provide  relatively  direct  evidence  about  the 
extent  to  which  the  reinforcers  assessed  by  the  Occupational  Values  Questionnaire  can 
differentiate  among  occupations.  The  results  suggest  that  in  the  current  sample  the  reliability  of 
occupation  mean  ratings  of  individual  reinforcers  is  low,  and  that  reasonably  reliable  ratings  at 
the  level  of  individual  reinforcers  would  require  at  least  30  respondents  per  occupation. 

However,  the  evidence  considering  the  reliability  of  the  questionnaire  as  a whole  is  more 
optimistic.  Table  10-5  presents  an  interrater  agreement  of  & = .60  for  the  questionnaire  as  a 
whole  with  a harmonic  mean  of  k = 9.69  respondents  per  occupation.  This  table  also  shows  a 
very  respectable  estimate  of  what  the  interrater  agreement  would  be  if  there  were  30  respondents 
per  occupation  foo  = .82).  This,  by  itself,  is  evidence  that  if  the  Occupational  Values 
Questionnaire  were  supported  by  30  respondents  per  occupation,  it  could  reliably  differentiate 
among  occupations. 

Some  of  the  analyses  address  the  validity  of  the  Occupational  Values  Questionnaire  in 
terms  of  the  similarities  and  differences  among  occupations.  Examination  of  the  occupation 
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means  on  the  21  reinforcers  across  six  varied  occupations  (see  Table  10-12)  indicates  that  some 
reinforcers  show  little  variation  across  the  six  occupations;  however,  others  do.  Social  Service 
(Descriptor  #15),  Social  Status  (Descriptor  #16),  and  Activity  (Descriptor  #3)  are  examples  of 
reinforcers  that  vary  in  manner  consistent  with  expected  similarities  and  differences  among  the 
examined  occupations.  The  results  of  the  discriminant  analysis  indicate  that  for.  this  sample  the 
Occupational  V alues  Questionnaire  is  not  strong  in  terms  of  differentiating  among  occupations. 
However,  the  six  reinforcers  that  appear  to  be  the  best  at  discriminating  among  the  36 
occupations  are  (1)  Activity  (Descriptor  #3),  (2)  Social  Status  (Descriptor  #16),  (3)  Authority 
(Descriptor  #5),  (4)  Advancement  (Descriptor  #4),  (5)  Social  Service  (Descriptor  #15)  and  (6) 
Security  (Descriptor  #14),  respectively. 

Another  group  of  analyses  address  the  validity  of  the  Occupational  Values  Questionnaire 
in  terms  of  the  observed  relationships  among  the  ratings.  Tables  10-6  and  10-7  show  that 
aggregate  scores  based  on  the  six  hypothesized  dimensions  presented  in  Figure  10-4  do  not  fit  the 
current  data.  The  reliability  of  the  aggregate  scores  is  slightly  lower  than  the  reliability  of  the 
occupation  mean  ratings  on  the  21  reinforcers  (see  Tables  10-5  and  10-7).  Examination  of  the 
correlations  among  reinforcer  ratings  and  the  results  of  an  attempted  confirmatoiy  analysis  also 
suggest  that  the  reinforcers,  as  rated  by  the  Occupational  Values  Questionnaire,  do  not  fit  the 
hypothesized  structure.  As  discussed  previously,  these  results  are  not  surprising  for  a number  of 
reasons  not  the  least  of  which  is  that  the  hypothesized  structure  is  not  based  on  analyses  of  an 
occupation  assessment,  but  on  analyses  of  a person  measure  (i.e.,  the  MIQ).  However,  despite  the 
failure  of  the  hypothesized  structure,  the  exploratory  principal  components  analysis  suggests  a 
structure  that  is  interpretable  and  reasonably  consistent  with  theory  (see  Table  10-11). 
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Table  10-17  represents  an  alternative  method  of  examining  the  reliability  and  validity  of 
the  Occupational  Values  Questionnaire.  Reliability  in  this  context  is  defined  as  the  extent  to 
which  the  mean  ratings  for  one  group  of  respondents  from  a particular  occupation  are  related  to 
the  mean  ratings  for  another  group  of  respondents  from  the  same  occupation.  Here  the  reliability 
of  the  Occupational  Values  Questionnaire,  supported  by  30  respondents  per  occupation,  is  strong. 
The  results  presented  in  Table  10-17  also  support  the  hypothesis  that  when  this  questionnaire  is 
completed  by  30  respondents  per  occupation,  it  can  recognize  differences  and  similarities  in 
occupational  reinforcer  profiles  across  occupations. 

Summary 

Research  and  theory  support  the  hypothesis  that  the  constructs  measured  by  the 
Occupational  Values  Questionnaire  are  relevant  to  the  description  of  occupations.  Further,  this 
questionnaire  is  practical  and  useable  in  the  context  of  0*NET.  It  is  simple  and  short  relative  to 
the  other  questionnaires.  The  results  of  this  study  also  provide  favorable  evidence  regarding  the 
ability  of  the  questionnaire  as  a whole  to  reliably  differentiate  among  occupations  when  there  are 
30  respondents  per  occupation.  This  successful  differentiation  among  occupations  minimizes  the 
concern  that  not  using  a forced  choice  rating  format  would  result  in  too  little  variation  in  ratings 
across  occupations. 

However,  a concern  remains.  What  is  the  structure  underlying  the  relationships  among  the 
reinforcers  as  they  are  measured  by  the  Occupational  Values  Questionnaire,  and  does  it  make 
sense?  However,  the  observed  factor  structure,  as  describe  by  the  exploratory  principal 
components  analysis,  is  interpretable  by  itself  and  in  the  context  of  the  hypothesized  structure. 
Recommendations 
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As  was  true  for  the  measurement  of  interests  using  the  Holland  codes,  the  current 
evidence  supports  the  inclusion  of  the  Occupational  Values  Questionnaire  in  the  0*NET  content 
model.  If  the  subsequent  data  collection  plans  include  approximately  30  raters  per  occupation, 
this  questionnaire  should  be  maintained  as  part  of  the  system.  However,  additional  studies,  with 
larger  numbers  of  occupations  and  respondents  per  occupation  would  allow  a more  definitive 
evaluation  of  the  extent  to  which  the  questionnaire  can  reliably  differentiate  among  occupations. 
Additionally,  comparison  of  these  ratings  to  ratings  in  other  domains  of  the  content  model  would 
enhance  the  evaluation  of  the  Occupational  Values  Questionnaire. 

The  results  suggest  that  some  of  the  individual  items  (i.e.,  reinforcer  statements)  could 
not  reliably  differentiate  among  occupations,  even  if  there  were  30  respondents  per  occupation. 
This  could  be  interpreted  as  suggesting  that  individual  items  showing  low  reliabilities  should  be 
eliminated  from  the  questionnaire.  However,  it  is  recommended  that,  for  now,  all  21  items  be 
retained  in  the  questionnaire  because  further  studies  including  more  occupations  and  more 
respondents  per  occupation  may  provide  more  favorable  item  level  reliability  evidence. 
Additionally,  the  Occupational  Values  Questionnaire  would  no  longer  represent  a complete 
taxonomy  of  vocational  values  if  items  were  removed. 

Final  Comments 

The  evidence  for  the  Holland  codes  and  the  Occupational  Values  Questionnaire  as 
assessments  of  interests  and  values  is  favorable,  and  information  in  these  domains  is  an 
important  part  of  occupation  description.  “The  primary  use  of  vocational  interest  and  value 
information  has  been  and  will  likely  continue  to  be  helping  people  make  vocational  choices 
(Dawis,  1991;  Holland,  1976)”  (Sager,  1995,  p.  11-16).  After  all,  matching  people  to  occupations 
is  one  of  0*NET’s  primary  goals.  It  is  also  relevant  to  note  that  vocational  and  career  counselors 
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have  been  frequent  purchasers  of  the  DOT  in  the  past  (Miller,  Treiman,  Cain,  & Roos,  1980). 
Including  interest  and  values  information  in  the  0*NET  content  model  would  provide  substantial 
support  for  the  work  of  vocational  and  career  counselors. 


(k: 


o 

ERIC 


10-32  Sager 


References 

Borgen,  F.H.,  Weiss,  D.J.,  Tinsley,  H.E.A.,  Dawis,  R.V.,  & Lofquist,  L.H.  (1968).  The 
measurement  of  occupational  reinforcer  patterns.  Minnesota  Studies  in  Vocational 
Rehabilitation,  XXV.  Minneapolis:  University  of  Minnesota 

Dawis,  R.V.  (1991).  Vocational  interests,  values,  and  preferences.  In  M.  D.  Dunnette  and 
L.  M.  Hough  (Eds.),  Handbook  of  industrial  and  organizational  psychology.  Vol.  2 of  2nd 
Edition.  Palo  Alto,  CA:  Consulting  Psychologists  Press. 

Dawis,  R.V.,  & Lofquist,  L.H.  (1984).  A psychological  theory  of  work  adjustment:  An 
individual-differences  model  and  its  applications.  Minneapolis:  University  of  Minnesota  Press. 

Gay,  E.G.,  Weiss,  D.J.,  Hendel,  D.D.,  Dawis,  R.V.,  & Lofquist,  L.H.  (1971).  Manual  for 
the  Minnesota  importance  questionnaire.  Minnesota  Studies  in  Vocational  Rehabilitation. 
XXVHI.  Minneapolis:  University  of  Minnesota. 

Gottfredson,  G.  D.,  & Holland,  J.  L.  (1989).  Dictionary  of  Holland  occupational  codes 
(2nd  ed.).  Odessa,  FL:  Psychological  Assessment  Resources. 

Gottfredson,  G.  D.,  & Holland,  J.  L.  (in  press).  Dictionary  of  Holland  occupational  codes 
(3rd  ed.).  Odessa,  FL:  Psychological  Assessment  Resources. 

Gottfredson,  G.  D.,  Holland,  J.  L.,  Ogawa,  D.  K.  (1982).  Dictionary  of  Holland 
occupational  codes  . Palo  Alto,  CA:  Consulting  Psychologists  Press. 

Hakel,  D.T.  (1986).  Personnel  selection  and  placement.  Annual  Review  of  Psychology. 
37,  351-380. 

Hansen,  J.C.,  & Campbell,  D.P.  (1985).  Manual  for  the  Strong  interest  inventory:  Form 
T325  of  the  Strong  vocational  interest  blanks.  Palo  Alto,  CA:  Consulting  Psychologists  Press. 


1073 


Occupational  Interests  and  Values  10-33 


Holland,  J.  L.  (1976).  Vocational  preferences.  In  M.  D.  Dunnette  (Ed.),  Handbook  of 
industrial  and  organizational  psychology  (pp.  521-570).  Chicago:  Rand  McNally. 

Joreskog,  K.  G.,  & Sorbom,  D.  (1993).  LISREL  (Version  8.03)  [Computer  Software], 
Chicago:  Scientific  Software  International. 

Miller,  A.  R.,  Treiman,  D.  J.,  Cain,  P.  S.,  & Roos,  P.  A.  (Eds.).  (1980).  Work,  jobs,  and 
occupations:  A critical  review  of  the  Dictionary  of  Occupational  Titles.  Washington,  DC: 
National  Academy  Press. 

National  Crosswalk  Service  Center.  (1993).  NOICC  Master  Crosswalk  (Version  4.0) 
[Electronic  data  file  and  documentation].  Des  Moine,  IA:  Author  [Producer  and  Distributor], 
Peterson,  N.  G.,  Mumford,  M.  D.,  Borman,  W.  C.,  Jeanneret,  P.  R.,  & Fleishman,  E.  A. 
(Eds.).  (1995).  Development  of  prototype  Occupational  Information  Network  (Q*NET)  content 
model  (Vols.  1-2).  Salt  Lake  City,  UT:  Utah  Department  of  Employment  Security. 

Sager,  C.  E.  (1995).  Occupational  Values  and  Interests.  In  N.  G.  Peterson,  M.  D. 
Mumford,  W.  C.  Borman,  P.  R.  Jeanneret,  & E.  A.  Fleishman  (Eds.),  Development  of  prototype 
Occupational  Information  Network  (Q*NET)  content  model  (Vols.  1-2).  Salt  Lake  City,  UT: 

Utah  Department  of  Employment  Security. 

Shrout,  P.  E.,  & Fleiss,  J.  L.,  (1979).  Intraclass  correlations:  Uses  in  assessing  rater 
reliability.  Psychological  Bulletin.  86. 420-428. 

U.S.  Department  of  Labor.  (1979).  Guide  for  occupational  exploration.  Washington,  DC: 

Author. 

Weiss,  D.J.,  Dawis,  R.V.,  England,  G.W.,  and  Lofquist,  L.H.  (1964).  The  measurement  of 
vocational  needs.  Minnesota  Studies  in  Vocational  Rehabilitation.  XVI.  Minneapolis:  University 


of  Minnesota. 


Figure  10-1 

Titles  And  Descriptions  of  the  Strong  Interest  Inventory  General  Occupational  Themes  (i.e., 
Holland  Taxonomy  of  Interests) 


Theme 

Description 

Realistic 

People  scoring  high  here  usually  are  rugged,  robust,  practical,  physically 
strong;  they  usually  have  good  physical  skills,  but  sometimes  have  trouble 
expressing  themselves  or  in  communicating  their  feelings  to  others. 

Investigative 

This  theme  centers  around  science  and  scientific  activities.  Extremes  of  this 
type  are  task-oriented;  they  are  not  particularly  interested  in  working  around 
other  people. 

Artistic 

The  extreme  type  here  is  artistically  oriented,  and  likes  to  work  in  artistic 
settings  that  offer  many  opportunities  for  self-expression. 

Social 

The  pure  type  here  is  sociable,  responsible,  humanistic,  and  concerned  with 
the  welfare  of  others. 

Enterprising 

The  extreme  type  of  this  theme  has  a great  facility  with  words,  especially  in 
selling,  dominating,  and  leading;  frequently  these  people  are  in  sales  work. 

Conventional 

Extremes  of  this  type  prefer  the  highly  ordered  activities,  both  verbal  and 
numerical,  that  characterize  office  work. 

Note:  This  table  contains  the  titles  of  the  General  Themes  from  the  SII  that  are  based  on 
Holland's  taxonomy  and  a brief  description  of  each;  the  descriptions  contain  only  a part  of 
language  used  to  describe  them  on  pages  14  and  15  of  the  Manual  for  the  Strong  Interest 
Inventory  (Hansen  & Campbell,  1985). 
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Figure  10-2 

Q*NET  Codes.  Titles,  and  Holland  Codes  for  Six  Occupations 


0*NET 

Occupation  Code 

Occupation  Title 

Holland 

Code3 

19005 

General  Managers  & Top  Executives 

ESR 

25105 

Computer  Programmers 

IRE 

32502 

Registered  Nurses, 

SIE 

63014 

Police  Patrol  Officers 

SER 

67005 

Janitors  and  Cleaners** 

REC 

85132 

Maintenance  Repairers,  General  Utility 

RIS 

Note.  The  Holland  Occupational  Codes  and  explanatory  text  included  in  this  document  are 
adapted  and  reproduced  by  special  permission  of  the  Publisher,  Psychological  Assessment 
Resources,  Inc.,  Odessa,  FL  33556,  from  the  Dictionary  of  Holland  Occupational  Codes.  Second 
Edition  by  Gary  D.  Gottfredson,  Ph.D.,  and  John  L.  Holland,  Ph.D.,  Copyright,  1982, 1989. 
Further  reproduction  for  any  purpose  or  by  any  means  is  prohibited  without  the  prior  written 
permission  of  the  Publisher. 

aR  = Realistic;  I = Investigative;  A = Artistic;  S = Social;  E = Enterprising;  C = Conventional. 
TTie  full  tide  for  this  job  is  “Janitors  and  Cleaners,  Except  Maids  and  Housekeeping.” 
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Figure  10-3 

Minnesota  Job  Description  Questionnaire’s  Twentv-One  Reinforcers  and  Their  Associated 
Statements  in  the  Occupational  Values  Questionnaire 


Reinforcer 

Reinforcer  Statement 

Each  statement  begins  with  “Workers  on  this  job  ...” 

1.  Ability  Utilization 

make  use  of  their  individual  abilities. 

2.  Achievement 

get  a feeling  of  accomplishment. 

3.  Activity 

are  busy  all  the  time. 

4.  Advancement 

have  opportunities  for  advancement. 

5.  Authority 

give  directions  and  instructions  to  others. 

6.  Company  Policies 

are  treated  fairly  by  the  company. 

7.  Compensation 

are  well  paid  in  comparison  with  other  workers. 

8.  Co-workers 

have  co-workers  who  are  easy  to  get  along  with. 

9.  Creativity 

try  out  their  own  ideas. 

10.  Independence 

do  their  work  alone. 

1 1.  Moral  Values 

are  never  pressured  to  do  things  that  go  against  their  sense  of 
right  and  wrong. 

12.  Recognition 

receive  recognition  for  the  work  they  do. 

13.  Responsibility 

make  decisions  on  their  own. 

14.  Security 

have  steady  employment. 

15.  Social  Service 

have  work  where  they  do  things  for  other  people. 

16.  Social  Status 

are  looked  up  to  by  others  in  their  company  and  their 
community. 

17.  Supervision  - Human 
Relations 

have  supervisors  who  back  up  their  workers  with 
management. 

18.  Supervision  - Technical 

have  supervisors  who  train  their  workers  well. 

19.  Variety 

have  something  different  to  do  every  day. 

20.  Working  Conditions 

have  good  working  conditions. 

21.  Autonomy 

plan  their  work  with  little  supervision. 

Note.  The  reinforcers  are  from  page  41  and  most  of  the  statements  are  from  the  Minnesota  Job 
Description  Questionnaire  (MJDQ)  in  Appendix  A of  A Psychological  Theory  of  Work 
Adjustment  by  Dawis  and  Lofquist  (1984).  However,  statements  5, 6, 8,  1 1,  16, 17,  and  18  were 
modified  in  the  Occupational  Values  Questionnaire  to  modernize  the  language. 


Figure  10-4 

The  Twenty-One  Reinforcers  From  the  Minnesota  Information  Questionnaire  and  Their 
Associated  Higher  Order  Dimensions 


1 Higher  Order  Dimension 

Reinforcer 

Achievement 

Ability  Utilization 

Achievement 

Comfort 

Activity 

Independence 

Variety 

Compensation 

Security 

Working  Conditions 

Status 

Advancement 

Recognition 

Authority 

Social  Status 

Altruism 

Co-workers 

Social  Service 

Moral  Values 

Safety 

Company  Policies 

Supervision,  Human  Relations 

Supervision,  Technical 

Autonomy 

Creativity 

Responsibility 

Autonomy 

Note:  The  information  in  this  table  is  abstracted  from  a table  on  page  849  of  Dawis  (1991). 


Figure  10-5 

Thirty-Six  Occupations  With  Four  or  More  Incumbents  Completing  the  Occupational  Values 
Questionnaire 


Occupation  Code 

Occupation  Title 

Number  of 
Respondents 

15005 

Education  Administrators 

11 

19005 

General  Managers  & Top  Executives 

43 

21108 

Loan  Officers  & Counselors 

6 

22135 

Mechanical  Engineers 

11 

25105 

Computer  Programmers 

7 

31303 

Teachers,  Preschool 

6 

31305 

Teachers,  Elementary  School 

13 

32502 

Registered  Nurses 

26 

32902 

Medical  & Clinical  Laboratory  Technologists 

7 

49008 

Salespersons,  Except  Scientific  & Retail 

14 

49011 

Salespersons,  Retail 

21 

49021 

Stock  Clerks,  Sales  Floor 

13 

49023 

Cashiers 

20 

51002 

First  Line  Supervisors,  Clerical/Administrative 

59 

53102 

Tellers 

4 

53311 

Insurance  Claims  Clerks 

7 

53905 

Teachers’  Aides  & Assistants,  Clerical 

9 

55108 

Secretaries,  Except  Legal  & Medical 

67 

55305 

Receptionists  & Information  Clerks 

6 

55338 

Bookkeeping,  Accounting,  & Auditing  Clerks 

27 

55347 

General  Office  Clerks 

92 

61005 

Police  & Detective  Supervisors 

13 

63014 

Police  Patrol  Officers 

24 

65008 

Waiters  & Waitresses 

11 

65026 

Cooks,  Restaurant 

5 

65038 

Food  Preparation  Workers 

31 

66005 

Medical  Assistants 

4 

66008 

Nursing  Aides,  Orderlies,  & Attendants 

21 

67005 

Janitors  & Cleaners 

29 

85119 

All  Other  Machinery  Maintenance  Mechanics 

4 

85132 

Maintenance  Repairers,  General  Utility 

26 

87902 

Earth  Drillers,  Except  Oil  & Gas 

5 

89108 

Machinists 

4 

92974 

Packaging  & Filing  Machine  Operators 

15 

97102 

Truck  Drivers,  Heavy  or  Tractor-Trailer 

9 

97111 

Bus  Drivers,  Schools 

11 

Table  10-1 

Descriptive  Statistics  Across  All  Occupations  and  Reliability  Estimates  for  Rated  Differences 
Between  Occupations:  Occupational  Values  Agreement  Ratines 


Agreement  Ratings 


Descriptor 

M 

SD 

SEMa 

Ikb 

1 . Ability  Utilization 

4.04 

0.35 

0.20 

0.67 

2.  Achievement 

3.81 

0.34 

0.24 

0.49 

3.  Activity 

4.00 

0.54 

0.25 

0.79 

4.  Advancement 

3.17 

0.54 

0.33 

0.63 

5.  Authority 

3.86 

0.52 

0.24 

0.78 

6.  Company  Policies 

3.47 

0.37 

0.29 

0.38 

7.  Compensation 

3.11 

0.47 

0.31 

0.55 

8.  Co-workers 

3.76 

0.31 

0.26 

0.31 

9.  Creativity 

3.53 

0.47 

0.24 

0.73 

10.  Independence 

3.10 

0.47 

0.28 

0.65 

11.  Moral  Values 

3.32 

0.34 

0.32 

0.10 

12.  Recognition 

3.18 

0.33 

0.28 

0.28 

13.  Responsibility 

3.62 

0.36 

0.24 

0.55 

14.  Security 

4.03 

0.33 

0.21 

0.59 

15.  Social  Service 

4.12 

0.37 

0.21 

0.67 

16.  Social  Status 

3.23 

0.44 

0.21 

0.77 

17.  Supervision  - Human  Relations 

4.48 

0.37 

0.31 

0.31 

18.  Supervision  - Technical 

3.17 

0.32 

0.29 

0.19 

19.  Variety 

3.46 

0.50 

0.27 

0.72 

20.  Working  Conditions 

3.71 

0.31 

0.22 

0.48 

21.  Autonomy 

3.81 

0.38 

0.25 

0.55 

Note.  Statistics  are  based  on  36  occupations  with  Occupational  Values  Questionnaire  responses 
from  at  least  4 incumbents  (mean  number  of  incumbents  = 18.92,  median  =12,  harmonic  mean 
9.69). 

“This  estimate  of  the  standard  error  of  measurement  was  calculated  as  SEM  = SD*  ^/(1-n) . 

‘’This  estimate  of  reliability  was  obtained  by  calculating  the  intraclass  correlation  for  k ratings 
across  occupations:  ICC(l.k)  = HBMS-WMS1/BMS  (Shrout  & Reiss,  1979),  where  k is  the 
harmonic  mean  of  the  number  of  ratings  provided  on  each  occupation. 
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Table  10-2 

Reliability  of  Rated  Differences  Between  Occupations  Considering  Varying  Numbers  of  Raters: 
Occupational  Values 

Numbers  of  Raters 
on  Each  Descriptor 


Descriptor jj3 


1.  Ability  Utilization 

18 

86 

2.  Achievement 

09 

75 

3.  Activity 

28 

92 

4.  Advancement 

15 

84 

5.  Authority 

27 

92 

6.  Company  Policies 

06 

65 

7.  Compensation 

11 

79 

8.  Co-workers 

04 

59 

9.  Creativity 

22 

89 

1 Independence 

16 

85 

11.  Moral  Values 

01 

26 

12.  Recognition 

04 

54 

13.  Responsibility 

11 

79 

14.  Security 

13 

82 

15.  Social  Service 

17 

86 

16.  Status 

25 

91 

17.  Supervision  - Human  Relations 

04 

58 

18.  Supervision  - Technical 

02 

42 

19.  Variety 

21 

89 

20.  Working  Conditions 

09 

74 

21.  Autonomy 

11 

79 

Note.  Statistics  are  based  on  36  occupations  with  Occupational  Values  questionnaire  responses 
from  at  least  4 incumbents  (mean  number  of  incumbents  = 18.92,  median  = 12,  harmonic  mean  = 
9.69).  Decimals  are  omitted. 

“Single  rater  estimates  of  reliability  were  obtained  by  calculating  the  intraclass  correlation  for 
single  judge  ratings  across  occupations:  ICC(1,1)  = rBMS-WMS1/rBMS+(k-lVWMSl  (Shrout  & 
Reiss,  1979),  where  k is  the  harmonic  mean  of  the  number  of  ratings  provided  on  each 
occupation. 

^Estimates  of  reliability  for  30  raters  were  obtained  by  applying  the  Spearman-Brown  correction 
formula  to  the  single  rater  reliability  estimates. 


Table  10-4 
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Variation  on  the  Agreement  Scale:  Occupational  Values 

eracuons  as 
MS 

ources  oi 
F 

Source  of  Variation 

SS 

df 

Occupations 

332.05 

35 

9.49 

2.37* 

S(Occupations) 

2584.07 

645 

4.01 

Descriptor 

801.08 

20 

40.05 

58.90* 

Descriptor  x Occupations 

1189.01 

700 

1.70 

2.50* 

Descriptor  x S(Occupations) 

8772.19 

12900 

0.68 

Note.  Occupations  are  treated  as  random,  between-subjects  effects,  while 
descriptors  are  treated  as  fixed,  within-subjects  effects. 

* p<.05 
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Table  10-5 

Interrater  Agreement  Coefficients  for  Each  Scale  Type:  Occupational  Values 

Number  of  Raters  on  Each  Variable 

Scale  Type ]jc Ii r3o 

Agreement .60 .13 .82 

Note.  Interrater  agreement  coefficient  estimates  are  based  on  36  occupations  with  Occupational 
Values  questionnaire  responses  from  at  least  4 incumbents  (mean  number  of  incumbents  = 18.92, 
median  = 12,  harmonic  mean  = 9.69).  Full  sample  interrater  agreement  coefficients  (&)  were 
obtained  by  considering  the  “Descriptor  x Occupations”  terms  from  Table  10-4  as  true  variance. 
Error  variance  was  defined  as  the  “Descriptor  x S(Occupations)”  term.  Estimates  of  reliability  for 
1 and  30  raters  were  obtained  by  applying  the  Spearman-Brown  correction  formula  to  the  k rater 
reliability  estimates,  where  k is  the  harmonic  mean  x>f  the  number  of  raters  for  each  occupation. 


Table  10-6 

Analysis  of  Variance  of  Aggregate  Descriptor.  Occupation,  and  Relevant  Interactions  as  Sources 
of  Variation  on  Agreement  Scale:  Occupational  Values 


Source  of  Variation 

SS 

df 

MS 

F 

Occupations 

103.00 

35 

2.94 

2.42* 

S (Occupations) 

785.75 

645 

1.22 

Aggregate 

82.38 

5 

16.48 

65.53* 

Aggregate  x Occupations 

103.45 

175 

0.59 

2.53* 

Aggregate  x S (Occupations) 

810.90 

3225 

0.25 

Note.  Occupations  are  treated  as  random,  between-subjects  effects,  while  aggregate  descriptors 
are  treated  as  fixed,  within-subjects  effects. 

*P<.05 
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Table  10-7 

Interrater  Agreement  Coefficients  for  Aggregate  Descriptors  for  Agreement  Ratings:  Occupational 


Number  of  Raters  on  Each  Variable 

TyPe a _ri r30 

Agreement 57 j2  81 

Note^  Interrater  agreement  coefficient  estimates  are  based  on  36  occupations  with  Occupational 
Values  Questionnaire  responses  from  at  least  4 incumbents  (mean  nurtiber  of  incumbents  = 1 8.92, 
median  =12,  harmonic  mean  = 9.69).  Full  sample  interrater  agreement  coefficients  (& ) were 
obtained  by  considering  the  “Aggregate  x Occupations”  terms  from  Table  10-6  as  true  variance. 
Error  variance  was  defined  as  the  “Aggregate  x S(Occupations)”  term.  Estimates  of  reliability  for  1 
and  30  raters  were  obtained  by  applying  the  Spearman-Brown  correction  formula  to  the  k rater 
reliability  estimates,  where  k is  the  harmonic  mean  of  the  number  of  raters  for  each  occupation. 
Decimals  are  omitted. 
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Table  10-11 

Principal  Components  Analysis  Pattern  Matrix  for  Agreement  Scale:  Occupational  Values 


Factor 


DescriDtor 

FI 

F2 

F3 

F4 

F5 

Communalitv 

1. 

Ability  Utilization 

.87 

.01 

.20 

.01 

-.05 

.81 

2. 

Achievement 

.84 

.25 

.05 

.17 

-.08 

.81 

3. 

Activity 

.20 

.61 

.13 

-.21 

-.32 

.58 

4. 

Advancement 

.09 

.34 

.04 

.73 

.36 

.78 

5. 

Authority 

.79 

.30 

.09 

-.08 

.26 

.79 

6. 

Company  Policies 

.28 

.46 

.48 

.40 

.15 

.70 

7. 

Compensation 

.05 

-.02 

.05 

.83 

-.10 

.71 

8. 

Co-workers 

-.05 

.13 

.76 

-.04 

.07 

.61 

9. 

Creativity 

.88 

.09 

.02 

-.02 

.11 

.79 

10. 

Independence 

-.16 

-.78 

.16 

-.10 

-.02 

.67 

11. 

Moral  Values 

.06 

-.03 

.66 

-.31 

-.27 

.60 

12. 

Recognition 

.38 

.15 

.62 

.41 

.15 

.74 

13. 

Responsibility 

.77 

-.11 

-.08 

-.02 

.26 

.68 

14. 

Security 

.19 

-.18 

-.11 

-.16 

.77 

.70 

15. 

Social  Service 

.15 

.21 

.19 

-.69 

.45 

.77 

16. 

Social  Status 

.37 

.17 

.14 

.17 

.77 

.81 

17. 

Supervision  - Human 
Relations 

.27 

.71 

.41 

.24 

.04 

.81 

18. 

Supervision  - Technical 

-.35 

.60 

.22 

-.12 

.40 

.71 

19. 

Variety 

.78 

.15 

.11 

-.11 

.36 

.79 

20. 

Working  Conditions 

.12 

.00 

.61 

.07 

.04 

.39 

21. 

Autonomy 

.71 

.05 

.30 

.29 

-.03 

.68 

Percent  Variance 

25 

12 

12 

12 

10 

Eigenvalue 

5.27 

2.52 

2.48 

2.44 

2.20 

Note.  N = 36.  The  correlation  matrix  was  based  on  means  calculated  at  the  occupation  level.  FI  = 
Individual  Accomplishment,  F2  = Structure,  F3  = Social  Comfort,  F4  = Career  Advancement,  F5 
= Stability.  These  loadings  are  based  on  an  orthogonal  varimax  rotation. 
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Table  10-13 

Rotated  Correlations  Between  Discriminating  Variables  and  Canonical  Discriminant  Functions 
for  Agreement  Scale:  Occupational  Values 

Functions 


Descriptor 

FI 

F2 

F3 

F4 

F5 

F6 

ZF2 

2 

T1 

1.  Ability  Utilization 

.13 

.07 

.12 

.08 

.03 

.02 

.04 

.14 

2.  Achievement 

.08 

.12 

.10 

.10 

.04 

.04 

.04 

.10 

3.  Activity 

.98 

-.01 

.06 

.06 

.00 

.05 

.97 

.21 

4.  Advancement 

.06 

.09 

.07 

.94 

.03 

.04 

.90 

.13 

5.  Authority 

.07 

.07 

.96 

.07 

.09 

.05 

.94 

.20 

6.  Company  Policies 

.01 

.11 

.06 

.14 

.04 

.10 

.05 

.08 

7.  Compensation 

.02 

.09 

.05 

.10 

.00 

.07 

.03 

.11 

8.  Co-workers 

.00 

.04 

.03 

.05 

.05 

.06 

.01 

.07 

9.  Creativity 

.01 

.07 

.12 

.08 

.07 

.05 

.03 

.17 

10.  Independence 

.04 

.03 

-.07 

-.05 

-.03 

.04 

.01 

.13 

11.  Moral  Values 

.02 

.00 

.00 

.07 

.04 

.06 

.01 

.06 

12.  Recognition 

-.01 

.14 

.02 

.16 

.04 

.07 

.05 

.07 

13.  Responsibility 

.06 

.05 

.08 

.03 

.06 

.09 

.03 

.11 

14.  Security 

.05 

.08 

.05 

.03 

.10 

.97 

.96 

.12 

15.  Social  Service 

.00 

.07 

.08 

.03 

.98 

.10 

.98 

.14 

16.  Social  Status 

-.02 

.93 

.08 

.09 

.08 

.09 

.89 

.19 

17.  Supervision  - Human 

.01 

.14 

.06 

.11 

.02 

.04 

.04 

.07 

Relations 

18.  Supervision  - Technical  .05 

.13 

.03 

.11 

-.01 

.03 

.03 

.06 

19.  Variety 

.07 

.09 

.07 

.10 

.06 

.01 

.03 

.16 

20.  Working  Conditions 

-.02 

.08 

.01 

.06 

.07 

.06 

.02 

.09 

21.  Autonomy 

.06 

.06 

.05 

.00 

.05 

.07 

.02 

.11 

Ec 

.58 

.57 

.50 

.46 

.39 

.37 

Percent  of  Variance 

19 

18 

12 

10 

7 

6 

Eigenvalue 

.50 

.47 

.34 

.27 

.18 

.16 

Note.  Statistics  are  based  on  36  occupations  with  Occupational  Values  questionnaire  responses 
from  at  least  4 incumbents  (mean  number  of  incumbents  = 18.92,  median  = 12,  harmonic  mean  = 
9.69).  FI  = Activity;  F2  = Social  Status;  F3  = Authority;  F4  = Advancement;  F5  = Social 
Service;  F6  = Security. 

XF  = Sum  of  squared  rotated  correlations  between  the  discriminating  variable  across  six 

discriminant  functions. 

2 

T)  = Variance  in  Level  Scale  ratings  accounted  for  by  occupations. 

The  statistics  “Rc,”  “Percent  of  Variance,”  and  “Eigenvalue,”  were  calculated  based  on  the 
unrotated  discriminant  functions. 
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Table  10-17 


Correlations  Among  Group  Mean  Ratings  Profiles  Between  and  Within 
Occupational  Values 


Three  Example  Occupations- 


Occ.  1 Occ.  1 Occ.  2 Occ.  2 Occ.  3 Occ.  3 
Group  la  Group  2 Group  1 Group  2 Group  1 Group  2 

Occ.  1 Group  1 — 


Occ.  1 Group  2 

86 



Occ.  2 Group  1 

69 

74 



Occ.  2 Group  2 

64 

70 

87 

Occ.  3 Group  1 

77 

84 

87 

84 

Occ.  3 Group  2 

74 

75 

85 

86 

No^  Within  each  occupation,  N = 60  raters  were  randomly  divided  in  half;  therefore  each  group’s 
profile  of  mean  ratings  is  based  on  N = 30  raters.  This  process  was  performed  10  times  resulting  in  10 
different  randomly  assigned  sets  of  six  groups  (i.e.,  2 groups  per  occupation).  A correlation  matrix  was 
calculated  for  each  set  of  six  group  profiles.  Each  element  of  this  table  contains  the  median  of  its 
associated  elements  across  all  10  correlation  matrices. 

Occ.  1 = First-Line  Supervisors  and  Managers/Supervisors;  Clerical  and  Administrative  Support 
Workers;  Occ.  2 = Secretaries  Except  Legal  and  Medical;  Occ.  3 = General  Office  Clerks. 

Decimals  are  omitted. 

The  mean  profile  for  this  group  is  based  on  only  n = 29  raters  because  the  sample  available  for  analysis 
included  only  n — 59  raters  of  this  occupation. 


Chapter  11 

Work  Style  Descriptors: 

Evidence  for  the  Reliability  and  Validity  of  the  Measures 

Walter  C.  Borman 
U.  Christean  Kubisiak 

Personnel  Decisions  Research  Institutes,  Incorporated 


In  this  chapter  and  elsewhere  in  the  0*NET  project,  we  use  the  term  work  styles  as  a 
label  for  this  domain.  The  domain  is  very  similar  to  personality,  but  to  emphasize  the  intention 
that  the  domain  contains  only  occupation-related  aspects  of  personality,  the  term  work  styles 
appears  most  appropriate. 

Work  styles  requirements  in  occupations  are  becoming  more  important  as  occupations 
and  organizations  undergo  what  is  proving  to  be  considerable  change.  Changes  include 
organizations  increasingly  employing  teams  to  do  work  (e.g.,  Guzzo  & Salas,  1995), 
organizational  citizenship  or  contextual  performance  being  more  often  considered  as  important 
(e.g.,  Borman  & Motowidlo,  1993;  Landy,  Shankster,  & Kohler,  1994),  service  jobs  becoming 
more  numerous  (e.g.,  Schneider,  1990),  and  person-organization  fit  between  employees  and  the 
values  and  “personality”  of  the  organization  increasingly  being  a focus  of  study  (e.g.,  Borman, 
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Hanson,  Hedge,  in'  press;  Schneider,  Goldstein,  & Smith,  1996).  Each  of  these  trends  is 
associated  with  increased  importance  for  work  styles.  To  be  successful,  teams  require  that  their 
members  have  certain  interpersonal  and  consensus-building  skills.  Contextual  performance  has 
as  antecedents  or  predictors  personality  constructs.  Most  service  jobs  require  strong  interpersonal 
skills.  And,  person-organization  fit  generally  focuses  on  the  personality,  motivational,  and  values 
domains.  Accordingly,  the  work  styles  is  an  important  part  of  the  0*NET  content  model. 

Background 

Taxonomy 

The  Borman,  McKee,  and  Schneider  (1995)  taxonomy  of  work  styles  is  intended  to 
provide  a comprehensive  summary  of  work  style  requirements  for  all  occupations  in  the  U.S. 
economy.  The  taxonomy  is  also  intended  to  be  efficient.  An  objective  was  to  represent  these 
occupational  requirements  with  a relatively  small  number  of  work  style  constructs.  As  with  other 
areas  in  the  content  model,  the  work  styles  taxonomy  is  hierarchical,  consisting  of  seven 
constructs  at  the  first  level  and  17  constructs  at  the  second  level. 

Briefly,  development  of  the  first  level  constructs  was  influenced  strongly  by  certain 
models  and  systems  in  the  personality  literature.  As  a point  of  departure,  the  five  factor  model 
(Goldberg,  1993;  Tupes  & Christal,  1992)  was  considered  carefully,  as  were  the  Hogans’  six 
dimension  system  for  the  Hogan  Personality  Inventory  (Hogan  & Hogan,  1992),  Hough’s  nine 
dimension  taxonomy  (Hough,  1992),  constructs  from  the  Occupational  Personality  Questionnaire 
(Saville  & Holdsworth,  1990),  and  dimensions  from  an  effort  to  develop  a job  analysis 
questionnaire  intended  to  measure  personality  requirements  of  jobs  (Guion,  1992).  We  also 
reviewed  factor  analytic  research  and  correlational  studies  in  the  area  of  personality  in  order  to 
decide  upon  an  appropriate  number  of  first  and  second  level  constructs,  to  assess  the  relative 
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independence  of  various  constructs,  and  to  better  understand  the  nature  of  constructs  in  our 
taxonomy. 

The  first  higher-order  construct  is  Achievement  Orientation.  This  construct  involves 
striving  for  competence  in  the  work  environment  and  valuing  hard  work,  persisting  despite 
obstacles,  setting  high  standards  for  performance,  and  wanting  to  get  ahead.  The  three 
subconstructs  for  the  second  level  taxonomy  are  Achievement/Effort  (Descriptor  #1),  Persistence 
(Descriptor  #2),  and  Initiative  (Descriptor  #3).  Achievement/Effort  (Descriptor  #1)  reflects 
setting  difficult  self-goals  and  expending  considerable  effort  toward  achieving  those  objectives. 
Persistence  (Descriptor  #2)  includes  the  overcoming  obstacles  element  of  Achievement 
Orientation,  and  Initiative  (Descriptor  #3)  reflects  the  notion  of  willingly  taking  on  additional 
responsibilities  and  challenges. 

The  second  higher-order  construct.  Social  Influence,  reflects  components  of  interpersonal 
impact,  persuasiveness,  and  energy.  Thus,  the  lower-order  constructs  are  Energy  (Descriptor  #4) 
and  Leadership  Orientation  (Descriptor  #5).  Energy  (Descriptor  #4)  represents  the  drive  and 
movement  toward  having  impact  in  a work  setting.  Leadership  Orientation  (Descriptor  #5)  also 
has  to  do  with  activating  the  potency  and  persuasiveness  necessary  to  influence  others  at  work. 

Interpersonal  Orientation  is  the  third  higher-order  construct.  This  reflects  aspects  of 
sensitivity  to  others  and  generally  getting  along  well  with  others  in  the  workplace.  Accordingly, 
the  subconstruct  Cooperation  (Descriptor  #6)  reflects  the  getting  along  well  component  of 
Interpersonal  Orientation.  Concern  for  Others  (Descriptor  #7)  represents  the  sensitivity  element, 
and  Social  Orientation  (Descriptor  #8)  is  the  affiliation,  liking  to  be  around  other  people  part  of 
the  higher-order  construct. 
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The  fourth  higher-order  construct  is  Adjustment,  being  calm,  composed,  and  rational 
even  under  stressful  circumstances.  This  also  involves  being  adaptable  to  changing  work 
situations.  Adjustment  thus  decomposes  into  the  second-level  constructs,  Self-Control 
(Descriptor  #7),  Stress  Tolerance  (Descriptor  #10),  and  Adaptability/Flexibility  (Descriptor  #1 1). 
The  first  lower-order  construct  reflects  the  evenness  of  mood,  controlling  anger  aspect  of 
Adjustment.  Stress  Tolerance  (Descriptor  #10)  brings  in  the  notion  of  constructively  and 
successfully  reacting  to  stressors.  And  Adaptability/Flexibility  (Descriptor  #11)  reflects 
effectively  responding  to  changes  in  the  workplace. 

The  fifth  higher-order  construct  is  Conscientiousness.  This  is  defined  as  being  careful, 
planful,  dependable,  disciplined,  as  well  as  trustworthy  and  accepting  of  authority.  The  first 
subconstruct  is  Dependability  (Descriptor  #12),  being  trustworthy  and  adhering  to  a strong  work 
ethic.  Attention  to  Detail  (Descriptor  #13)  covers  the  careful,  planful  elements  of  the  higher- 

order  construct.  Finally,  Integrity  (Descriptor  #14)  involves  being  honest  and  avoiding  unethical 
behavior. 

Independence  (Descriptor  #15)  is  the  sixth  higher-order  construct.  This  has  to  do  with 
performing  effectively  with  little  or  no  supervision  and  developing  one’s  own  ways  of 
succeeding  on  the  job.  This  construct  has  no  subconstructs. 

Finally,  the  seventh  higher-order  construct  is  Practical  Intelligence.  We  define  this 
construct  as  generating  useful  ideas  at  work  and  thinking  things  through  logically.  Accordingly, 
the  subconstructs  are  Innovation  (Descriptor  #16)  and  Analytical  Thinking  (Descriptor  #17).  The 
Innovation  (Descriptor  #16)  dimension  refers  to  being  creative  and  coming  up  with  new  ideas 
and  answers  to  work-related  problems.  Analytical  Thinking  (Descriptor  #17)  reflects  the 
analyzing  information  and  using  logic  to  address  work  problems  aspect  of  Practical  Intelligence. 
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Figure  11-1  describes  higher-order  constructs  and  presents  the  17  construct  lower-order 
system.  See  Borman,  McKee,  and  Schneider  (1995)  provide  a complete  description  of  work  style 
construct  development  and  a cross-walk  companion  of  this  taxonomy  with  other  personality 
taxonomies  and  dimensional  systems.  We  believe  that  the  work  styles  taxonomy  meets  the 
objectives  of  being  comprehensive  but  at  the  same  time  parsimonious.  Nonetheless,  at  this  point 
we  had  no  evidence  of  reliability  or  validity  for  the  work  styles  constructs.  The  next  sections  of 
the  chapter  begin  to  address  issues  of  reliability  and  validity. 

Sample  and  Measures 

As  mentioned  in  previous  chapters,  the  target  number  of  occupations  for  the  initial  data 
collection  was  80.  However,  the  analyses  to  be  described  involve  35  occupations  with  a 
minimum  of  four  incumbents  per  occupation.  Figure  11-2  lists  these  occupations  and  the  N for 
each.  The  number  of  incumbents  actually  ranged  from  four  to  40,  with  an  harmonic  mean  of  9.81 
per  occupation.  OAFC  analysts  did  not  provide  work  style  ratings. 

Regarding  the  work  style  survey  instrument,  level  and  importance  scales  were  developed. 
The  level  scales  are  7-point  scales  (1-7)  with  an  additional  not-relevant  option.  Three  behavioral 
anchors  were  also  developed,  at  the  high,  mid-range,  and  low  levels.  These  anchors  were 
reviewed  by  the  project  staff,  OAFC  staff,  and  persons  in  our  pilot  sample.  Revisions  were  made 
based  on  their  comments,  and  the  anchors  were  placed  opposite  the  6-7, 4,  and  1-2  scale  points, 
respectively.  The  level  scales  can  best  be  characterized  as  measuring  complexity.  The  importance 
scales  are  5-point  (1-5)  scales  with  the  verbal  anchors  Not  Important,  Somewhat  Important, 
Important,  Very  Important,  and  Extremely  Important,  respectively,  for  the  1 to  5 scale  points.  As 
with  the  other  domains,  for  each  descriptor,  the  level  question  was  asked  first  followed  by  the 
importance  question.  Figure  1 1-3  provides  an  example  level  and  importance  scale. 
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Results 

Descriptive  Statistics 

Table  11-1  contains  the  overall  means  across  the  35  occupations.  Also  shown  are  the 
standard  deviations  and  interrater  agreement  indices  based  on  9.81  raters  per  occupation,  and  the 
standard  errors  of  measurement.  Focusing  on  the  level  ratings,  the  means  are  high  and  the  range 
is  quite  restricted  for  some  of  the  descriptors.  Dependability  (Descriptor  #12)  is  most  noteworthy 
in  this  regard,  with  a mean  of  6.30  and  a standard  deviation  of  only  .36.  Attention  to  Detail 
(Descriptor  #13),  Cooperation  (Descriptor  #6),  Stress  Tolerance  (Descriptor  #10),  and  to  a lesser 
extent  Adaptability/Flexibility  (Descriptor  #11)  also  have  high  means  and  standard  deviations 
that  are  somewhat  restricted. 

The  pattern  of  means  and  standard  deviations  for  the  importance  ratings  is  similar  to  that 
experienced  with  the  level  ratings.  Means  for  Dependability  (Descriptor  #12),  Attention  to  Detail 
(Descriptor  #13),  and  Cooperation  (Descriptor  #6)  are  high  and  their  standard  deviations  are 
restricted.  The  mean  for  Integrity  (Descriptor  #14)  is  also  high  but  its  standard  deviation  is  not  so 
restricted. 

Reliability 

The  median  k-rater  reliability  is,  for  the  level  ratings,  .66,  with  Dependability  (Descriptor 
#12)  the  lowest  at  .15.  The  reliability  for  this  descriptor  is  probably  that  low  because  of  the 
serious  restriction-in-range.  Interrater  agreement  for  the  importance  ratings  are  comparable,  but 
slightly  lower  than  those  for  the  level  ratings.  The  median  k-rater  reliability  is  .64.  Again, 
Dependability  (Descriptor  #12)  has  the  lowest  reliability  (r  = .26). 

Table  11-2  displays  the  1 -rater  and  30-rater  reliabilities,  using  the  Spearman-Brown 
formula  to  make  the  estimates.  The  1 -rater  reliabilities  are  indeed  quite  low  for  both  the  level  and 
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the  importance  ratings.  However,  the  30-rater  reliabilities  are  almost  all  above  .75,  and  many  are 
above  .85.  If  we  were  to  attain  our  goal  of  getting  30  incumbents  per  occupation,  the  reliabilities 
for  work  styles  would  be  very  acceptable  with  the  exception  of  the  Dependability  (Descriptor  ' 
#12)  descriptor.  It  may  be  necessary  to  have  somewhat  higher  numbers  of  incumbents  completing 
the  work  styles  scales  compared  to  the  scales  in  some  of  the  other  domains  such  as  skills  and 
abilities.  Nonetheless,  we  have  demonstrated  that  it  is  possible  to  obtain  reasonably  reliable 
incumbents’  ratings  of  the  work  style  requirements  for  their  jobs.  To  our  knowledge,  incumbents 
have  not  heretofore  been  asked  to  make  work  style  requirements  judgments  about  jobs  and 
occupations.  This  study  indicates  that  it  is  feasible  to  gather  this  kind  of  information. 

Scoring 

As  mentioned  in  previous  chapters,  it  might  be  argued  that  including  not  relevant  (i.e., 
zero)  scores  in  the  computation  of  level  scale  reliabilities  is  inappropriate.  A similar  argument 
could  be  made  for  the  importance  scales  in  that  when  not  relevant  was  indicated  on  the  level 
scale,  a 1 (not  important)  rating  was  used  in  computing  the  interrater  agreement  of  the 
importance  scales.  To  address  this  possibility,  we  recalculated  reliabilities,  not  considering  the 
not  relevant  responses.  Additionally,  the  level  scale  reliabilities  were  recomputed  using  a simple 
relevant/not  relevant  coding  scheme. 

Interrater  agreement  coefficients  using  these  recodings  are  presented  in  Table  11-3.  First, 
the  coefficients  are  virtually  unchanged  for  the  level  scales,  when  ignoring  the  not  relevant 
response  (comparing  r*  and  rb).  For  the  importance  scales,  interrater  agreement  drops 
considerably  for  one  scale  (Independence  [Descriptor  #15]),  a small  amount  for  eight  of  the 
scales,  and  actually  improves  for  six  of  the  scales.  In  two  cases  the  coefficients  did  not  change.  In 
summary,  reliability  suffered  only  minimally  or  not  at  all  when  the  not  relevant  rescaling  was 
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employed  for  the  work  styles  descriptors.  Except  for  one  case  (Independence  [Descriptor  #15]), 
interrater  agreement  was  considerably  lower  when  level  scales  were  scored  relevant/not  relevant. 
Accordingly,  this  simple  scoring  method  does  not  appear  to  be  prudent. 

Overall,  the  full  scale  scoring  method  including  not  relevant  appears  preferable  and  was 
used  for  subsequent  analyses.  Considering  not  relevant  in  the  level  scale  scoring  adds 
information  that  for  the  most  part  enhances  the  reliability  of  the  importance  scales.  This  has 
intuitive  appeal  in  that  a not  relevant  rating  for  a work  style  seems  conceptually  quite  different 
from  a low  level  requirement  for  that  work  style,  and,  in  parallel,  being  of  no  importance  is 
different  from  being  of  low  importance. 

Analyses  of  Variance 

A summary,  across  descriptor  approach  to  examining  the  reliability  of  the  work  styles 
descriptor  scales  is  to  use  analysis  of  variance  (ANOVA).  Results  of  these  analyses  are  shown  in 
Tables  1 l-4a  and  1 l-4b  for  the  level  and  importance  scales,  respectively.  First,  for  both  types  of 
scales,  a significant  occupations  effect  indicates  that  work  styles  do  in  fact  discriminate  between 
occupations.  A highly  significant  descriptor  effect  shows  that  some  work  style  dimensions  are  on 
average  required  at  a higher  level  and  are  more  important  than  others.  Most  relevant,  the 
significant  descriptor  x occupation  interaction  provides  evidence  that  different  work  styles 
differentiate  between  different  occupations.  That  is,  different  patterns  of  work  style  requirements 
are  evident  for  different  occupations. 

The  intraclass  correlations  summarizing  the  interrater  agreement  from  the  ANOVAs  are 
in  Table  1 1-5.  The  k-rater  coefficients  are  .70  and  .67  for  the  level  and  importance  scales.  The 
30-rater  coefficients  are  near  .90.  Again,  this  indicates  that  interrater  agreement  is  quite  good, 
especially  if  the  original  target  of  30  raters  per  occupation  can  be  achieved. 
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Tables  1 l-6a  and  1 l-6b  present  ANOVA  results  for  the  aggregate  work  styles  scales,  that 
is,  for  mean  scores  computed  across  the  scales  combined  according  to  the  higher-order  7- 
dimension  system.  As  with  the  17-dimension  system,  the  desirable  occupations  and  descriptor  x 
occupations  effects  are  significant,  indicating  that  work  styles  can  in  general  differentiate 
between  occupations  and  that  different  combinations  of  work  styles  provide  the  differentiation 
depending  on  the  occupations  in  the  mix.  The  interrater  agreement  coefficients  of  .76  and  .73 
(Table  1 1-7)  indicate  that  at  the  aggregate  level,  this  agreement  is  reasonably  high.  When  30 
raters  can  provide  ratings,  interrater  agreement  is  expected  to  be  in  the  .90  range.  Accordingly, 
the  work  style  descriptors,  whether  at  the  individual  scale  or  aggregate  scale  level,  seem  to  have 
adequate  reliability. 

Descriptor  and  Scale  Relationships 

One  might  question  whether  data  from  the  two  scales,  level  and  importance,  are  largely 
redundant.  Table  11-8  presents  the  correlations  between  the  level  and  importance  scales, 
computed  two  different  ways,  for  each  dimension  across  the  35  occupations  and  averaged  over 
the  17  dimensions  and  for  each  occupation  across  the  17  dimensions  and  averaged  over  the  35 
occupations.  The  correlations  are  high  (.90  and  .93)  and  the  low  standard  deviations  indicate  that 
they  are  uniformly  high,  especially  by  occupation  across  the  17  dimensions.  These  results  suggest 
that  indeed  there  is  considerable  redundancy  across  the  level  and  importance  scales. 

Tables  1 l-9a  and  1 l-9b  present  the  correlations  between  dimensions  for  the  level  and 
importance  scales,  respectively.  The  correlations  are  at  the  occupation  level;  i.e.,  the  N of  each 
correlation  is  35.  Tables  1 l-10a  and  1 l-10b  present  the  same  correlations  but  at  the  individual 
respondent  level.  Because  our  primary  concern  in  the  research  is  to  identify  differences  between 
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occupations,  we  focus  our  discussion  on  the  occupation  level  data.  We  also  focus  our  discussion 
on  the  level  scales,  although  the  pattern  of  relationships  for  the  importance  scales  is  quite  similar. 

Correlations  in  Table  1 l-9a  make  good  intuitive  sense.  For  example,  correlations  among 
the  three  descriptors  that  comprise  the  higher  level  Achievement  construct  (Achievement/Effort 
[Descriptor  #1],  Persistence  [Descriptor  #2],  and  Initiative  [Descriptor  #3])  are  .78,  .65,  and  .79. 
Similarly,  the  Innovation  (Descriptor  #16)  and  Analytical  Thinking  (Descriptor  #17)  descriptors 
that  together  form  the  Practical  Intelligence  higher-order  factor  correlate  .81.  In  fact,  the  mean 
between  descriptor,  within  higher  construct  correlation  is  .69;  the  mean  between  descriptor, 
across  higher-order  construct  correlation  is  .42.  The  negative  correlations  are  also  where  they 
would  be  expected.  Independence  (Descriptor  #15)  correlates  -.35  with  Social  Orientation 
(Descriptor  #8)  and  -.18  with  Cooperation  (Descriptor  #6),  as  examples.  Thus,  the  work  styles 
scales  seem  to  provide  a meaningful  pattern  of  relationships. 

Factor  Structure 

Another  way  to  evaluate  the  patterns  of  relationships  between  work  style  dimensions  is  to 
factor  analyze  the  correlation  matrix  shown  in  Table  1 l-9a.  Accordingly,  we  conducted  a 
principal  components  factor  analysis  with  varimax  rotation.  The  2-7  factor  solutions  were 
examined  and  the  pattern  of  eigenvalues  along  with  interpretability  of  the  solutions  suggested 
that  the  3-factor  solution  best  summarized  the  correlation  matrix.  This  solution  is  depicted  in 
Table  1 1-1 1.  The  three  factors  accounted  for  75%  of  the  total  variance  for  ratings  on  the  level 
scale.  Also,  the  communalities  suggested  that  with  two  exceptions  the  dimensions  were  well 
represented  in  the  solution. 

The  first  factor  is  a strong  one  with  eight  of  the  17  dimensions  loading  substantially  on  it 
and  35%  of  the  variance  accounted  for.  We  labeled  the  factor  surgencv.  achievement,  high 
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activity  level  orientation.  Referring  to  the  higher-order  7-dimension  set.  Factor  1 combines  four 
of  these  dimensions:  Achievement  Orientation;  Social  Influence;  Independence;  and  Practical 
Intelligence.  Factor  2 was  called  people  orientation  and  accounted  for  29%  of  the  total  variance. 
Again,  focusing  on  the  higher-order  dimension  structure  to  aid  in  interpretation,  this  factor 
includes  all  of  the  Interpersonal  Orientation  descriptors  and  two  of  the  three  Adjustment 
descriptors.  The  third  factor,  accounting  for  12%  of  the  variance,  has  only  a single  high  loading 
descriptor,  Attention  to  Detail  (Descriptor  #13).  We  labeled  this  factor  detail  orientation.  Two  of 
the  Conscientiousness  descriptors  (Dependability  [Descriptor  #12]  and  Integrity  [Descriptor 
#14])  and  one  of  the  Adjustment  descriptors  (Adaptability/Flexibility  [Descriptor  #1 1])  are  not 
represented  in  the  3-factor  system.  However,  at  least  one  descriptor  from  each  of  the  seven 
higher-order  dimensions  loads  substantially  on  one  of  the  factors.  So,  from  that  perspective  the 
three  factors  reflect  well  the  entire  scope  of  the  work  styles  domain. 

Of  course,  it  must  be  remembered  that  this  factor  analysis  was  conducted  on  dat a 
representing  35  occupations.  The  stability  of  the  solution,  therefore,  may  not  be  high. 
Nonetheless,  the  three  factors  make  reasonably  good  conceptual  sense. 

We  also  attempted  to  conduct  confirmatory  factor  analyses  to  test  the  viability  of  the  7- 
dimension  higher-order  system.  Unfortunately,  the  solution  did  not  converge,  probably  because 
of  the  small  N in  comparison  to  the  number  of  variables  being  analyzed. 

Occupation  Differences 

One  of  the  ways  0*NET  will  be  of  use  in  actual  practice  is  to  generate  profiles  of 
occupational  requirements  on  the  0*NET  descriptors.  To  provide  an  initial  idea  of  the  usefulness 
of  profile  data  in  the  work  styles  domain,  Tables  1 l-12a  and  1 l-12b  contain  the  means  and 
standard  deviations  for  each  of  the  17  work  style  dimensions  on  level  and  importance  for  six 
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different  occupations  selected  to  reflect  very  different  types  of  employment:  (1)  General 
Managers  and  Top  Executives;  (2)  Computer  Programmers;  (3)  Registered  Nurses;  (4)  Police 
Patrol  Officers;  (5)  Janitors  and  Cleaners;  and  (6)  Maintenance  Repairers,  General  Utility.  « 
Table  1 l-12a,  depicting  the  level  means,  indicates  that  some  descriptors  have  relatively 
high  means  for  all  of  the  occupations.  For  example.  Dependability  (Descriptor  #12)  and 
Attention  to  Detail  (Descriptor  #13)  show  no  mean  ratings  below  5 on  the  7-point  level  scales. 
This  supports  the  ANOVA  finding  of  a significant  descriptor  effect.  However,  even  for  these 
descriptors  that  have  relatively  restricted  ranges  of  means,  there  is  some  differentiation  that 
makes  good  conceptual  sense.  For  example,  the  two  occupations  with  the  highest  Dependability 
(Descriptor  #12)  ratings  are  Patrol  Officers  and  Nurses.  These  same  two  occupations  plus 
Computer  Programmers  have  the  highest  Attention  to  Detail  (Descriptor  #13)  mean  ratings.  For 
the  remaining  descriptors,  the  differentiation  between  occupations  is  more  evident.  As  examples. 
Computer  Programmers  have  the  highest  rating  on  the  Analytical  Thinking  (Descriptor  #17)  level 
descriptor;  Janitors  have  the  lowest  rating  on  this  descriptor.  On  the  other  hand.  Janitors  have  a 
relatively  high  mean  rating  on  Energy  (Descriptor  #4).  General  Managers  have  the  highest  ratings 
on  Initiative  (Descriptor  #3)  and  Leadership  Orientation  (Descriptor  #5);  Nurses  the  highest 
ratings  on  Cooperation  (Descriptor  #6),  Concern  for  Others  (Descriptor  #7),  Social  Orientation 
(Descriptor  #8),  Adaptability,  and  (as  mentioned)  Attention  to  Detail  (Descriptor  #13);  and 
Computer  Programmers  the  highest  ratings  on  Independence  (Descriptor  #15)  (as  well  as 
Analytical  Thinking  [Descriptor  #17])  and  the  lowest  ratings  on  the  Cooperation  (Descriptor  #6), 
Concern  for  Others  (Descriptor  #7),  and  Social  Orientation  (Descriptor  #8)  descriptors.  In 
summary,  the  mean  level  ratings  are  consistent  with  the  nature  of  these  occupations.  The  work 
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style  level  scales  appear  to  provide  a meaningful  description  of  the  similarities  and  differences 
between  occupations. 

Some  of  the  standard  deviations  are  of  interest  as  well.  Several  of  the  highest  standard 
deviations  are  evident  with  the  Janitors  and  Cleaners.  For  example.  Leadership  Orientation 
(Descriptor  #5)  and  Analytical  Thinking  (Descriptor  #17)  have  high  standard  deviations  for  this 
group.  This  may  reflect  more  heterogeneity  in  the  occupations  within  occupation.  Some  of  the 
Janitors  sampled  may  have  supervisory  responsibilities,  for  example. 

We  have  concentrated  on  interpreting  the  level  scale  data.  However,  similar  conclusions 
may  be  drawn  from  Table  11-1 2b  and  the  importance  rating  data. 

A very  different  way  to  explore  occupation  differences  in  work  styles  is  to  compare 
patterns  of  work  style  ratings  within  occupation  to  the  patterns  across  occupations.  In  particular, 
a simple  way  to  address  occupation  differences  in  this  regard  is  to  identify  occupations  for  which 
we  have  a relatively  large  sample  size,  split  the  sample  in  half  within  each  of  the  occupations, 
and  correlate  the  mean  incumbents’  ratings  within  and  across  occupations.  The  hypothesis  is  that 
the  within  occupation  correlations  will  be  higher  than  the  across  occupation  correlations.  This 
analysis  was  carried  out  for  three  occupations:  first-line  supervisors;  secretaries  (except  legal  and 
medical);  and  general  office  clerks. 

Table  11-17  contains  these  correlations.  As  hypothesized,  in  general,  the  within 
occupation  relationships  are  higher  (M  within  = .94;  M across  = .79).  However,  the  clerk- 
secretary across  occupation  correlations  are  also  very  high.  These  occupations  would  be  expected 
to  do  very  similar  work,  so  this  result  is  not  surprising. 
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Discriminant  Analyses 

The  questions  might  be  asked,  how  well  do  the  work  style  descriptors  discriminate 

between  the  35  occupations  in  our  total  sample?  To  explore  this,  we  conducted  a discriminant 

function  analysis  on  the  level  scales.  This  analysis  evaluates  the  degree  of  differentiation  the 

work  styles  provide  in  general  and  the  variables  among  them  that  provide  the  most  (and  least) 

differentiation  between  the  35  occupations.  The  one  to  five  function  solutions  were  examined 

and  the  four-function  solution  was  selected  as  most  interpretable.  Table  1 1-13  presents  the 

rotated  correlations  between  the  discriminating  variables  and  the  four  canonical  discriminant 

functions.  These  variables  and  the  vectors  of  correlations  can  be  interpreted  as  the  loadings  on 

the  work  style  level  scales  that  maximally  differentiate  between  the  35  occupations.  Also 

% 

provided  in  the  table  are  T]2  coefficients  that  summarize  how  much  discriminating  variance  each 
work  style  level  scale  is  providing. 

The  first  function,  achievement/intelligence,  indicates  that  scales  from  the  higher  level 
Achievement  and  Practical  Intelligence  constructs  contribute  importantly  to  differentiating  these 
occupations.  The  two  Practical  Intelligence  descriptors  are  especially  important  in  differentiating 
between  occupations  according  to  the  T]2  results.  The  second  function  is  primarily  an  adjustment 
function,  the  third  can  be  interpreted  as  an  interpersonal  orientation  function,  and  the  fourth, 

energy,  is  defined  by  a positive  loading  on  Energy  (Descriptor  #4)  and  a negative  loading  on 
Attention  to  Detail  (Descriptor  #13). 

Overall,  the  work  style  level  scales  provided  some  differentiation  of  the  occupations, 
classifying  32%  of  incumbents  correctly  using  all  35  functions  and  24%  using  only  the  four 
functions  reported  in  Table  11-13.  The  T,2  vary  from  .22  down  to  .06.  As  expected.  Dependability 
(Descriptor  #12)  provided  the  least  differentiation,  probably  because  of  its  high  mean  and 
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restricted  range.  It  should  be  noted  that  the  work  style  scales’  T|2  are  not  in  general  quite  as  high 
as  those  obtained  in  some  of  the  other  domains,  such  as  skills.  Nonetheless,  these  findings  along 
with  the  occupation  differences  data  in  Table  1 l-12a,  indicate  that  the  work  styles  can  play  a role 
in  providing  information  about  job  requirements  that  helps  to  differentiate  between  occupations. 

Conclusions 

We  should  first  remind  the  reader  of  two  significant  limitations  to  the  research  conducted 
to  date.  The  sample  of  occupations  is  relatively  small  at  this  point,  and  the  number  of  incumbents 
per  occupation  providing  ratings  was  likewise  quite  small  in  some  cases.  Nonetheless,  research 
reported  here  indicated  that  job  incumbents  using  the  work  style  descriptor  scales  can  provide 
reliable  data.  Interrater  agreement  was  reasonably  high  for  both  the  level  and  importance  scales. 
If  the  target  of  30  incumbents  per  occupation  can  be  accomplished,  the  reliabilities  for  the  scales 
should  be  excellent.  It  was  noteworthy  that  the  level  scales  were  somewhat  more  reliable  than  the 
more  traditional  importance  scales. 

Regarding  relationships  between  the  two  scales,  correlations  were  high,  suggesting 
considerable  redundancy  in  level  and  importance  information.  It  should  be  kept  in  mind, 
however,  that  even  with  a high  correlation,  there  may  be  some  occupations  where  for  certain 
work  style  descriptors,  legitimate  and  substantial  differences  exist  between  level  and  importance. 
Nonetheless,  if  a scale-type  is  to  be  dropped,  we  would  suggest  eliminating  the  importance  scale. 
The  level  scales  are  slightly  more  reliable,  and  the  behavioral  anchors  attached  to  the  level  scales 
seemed  to  clarify  the  definition  of  each  work  style. 

Overall,  this  research  lends  support  for  the  work  styles  taxonomy  developed  by  Borman, 
McKee,  and  Schneider  (1995),  including  the  hierarchical  structure  they  proposed.  The  mean 
correlation  between-descriptor,  within-higher-order  construct  (e.g..  Achievement  Orientation  in 
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Figure  1 1-1)  is  .69;  the  mean  between-descriptor,  across-higher-order  construct  correlation  is  .42. 
Further,  other  between-descriptor  relationships  made  intuitive  sense,  and  a factor  analysis  of  the 
correlations  between  descriptors  yielded  a readily  interpretable  3-factor  solution. 

Perhaps  most  important  for  the  purposes  of  this  effort,  the  work  style  descriptors 
successfully  differentiated  between  the  35  occupations.  A discriminant  function  analysis  showed 
that  all  of  the  17  dimension(s)  played  a role  in  differentiating  at  least  some  of  the  occupations, 
although  Dependability  (Descriptor  #12)  was  marginal  in  this  regard.  The  taxonomy  is  primarily 
designed  to  describe  occupational  requirements  and  how  these  differ  across  occupations.  The 
work  styles  level  scales  seem  useful  in  this  role. 

Still,  the  question  might  be  asked,  can  we  drop  any  of  the  17  work  style  descriptors  based 
on  the  findings  presented?  We  would  argue  that  a very  important  consideration  here  is  the 
integrity  of  the  taxonomy.  Borman  et  al.  (1995)  developed  this  taxonomy  with  a strong 
theoretical  base.  It  is  comprehensive  and  yet  reasonably  parsimonious.  Nonetheless,  as 
mentioned.  Dependability  (Descriptor  #12)  performed  comparatively  poorly  in  the  research.  Its 
overall  mean  was  very  high  for  the  level  scale,  and  its  range  across  occupations  quite  restricted. 
Probably  because  of  this.  Dependability  (Descriptor  #12)  played  the  smallest  role  in 
differentiating  between  occupations  in  the  discriminant  function  analysis.  Thus,  this  descriptor  is 
a candidate  for  dropping,  or  perhaps  revising  to  create  more  differentiation  between  occupations. 
Of  course,  it  must  be  remembered  that  it  has  been  tried  on  only  35  occupations  at  this  point,  and 
Dependability  (Descriptor  #12)  may  do  better  at  differentiating  between  other  occupations.  It  is 
also  reasonable  to  ask  if  a work  style  taxonomy  without  the  Dependability  construct  is  sensible. 
Because  of  its  perceived  required  high  level  and  importance,  it  may  not  be  defensible  to  drop  it. 
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In  summary,  the  proposed  work  styles  taxonomy  received  considerable  support  from  this 
research.  The  descriptor  scales  measuring  the  taxonomy’s  constructs  yielded  reliable,  coherent, 
and  useful  data.  The  scales  appear  to  provide  an  adequate  basis  for  identifying  and  describing  the 
similarities  and  differences  across  occupations  in  the  domain  of  work  style  requirements. 
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Figure  1 1-1  (Continued) 

Descriptions  and  Definitions  of  Work  Styles 
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Figure  11-1  (Continued) 

Descriptions  and  Definitions  of  Work  Styles 
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Figure  11-1  (Continued) 

Descriptions  and  Definitions  of  Work  Styles 
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Figure  11-1  (Continued) 

Descriptions  and  Definitions  of  Work  Styles 
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Figure  11-2 

Thirty-five  Occupations  With  Four  or  More  Incumbents  Completing  the  Work  Styles 
Questionnaire 


Occupation 

Code 

Occupation  Title 

Number  of 
Respondents 

15005 

Education  Administrators 

11 

19005 

General  Managers  & Top  Executives 

44 

22135 

Mechanical  Engineers 

6 

25105 

Computer  Programmers 

8 

31305 

Teachers,  Elementary  School 

8 

31502 

Librarians,  Professional 

4 

32502 

Registered  Nurses 

34 

32902 

Medical  & Clinical  Laboratory  Technologists 

9 

32905 

Medical  & Clinical  Laboratory  Technicians 

5 

49008 

Salespersons,  Except  Scientific  & Retail 

7 

49011 

Salespersons,  Retail 

24 

49017 

Counter  & Rental  Clerks 

5 

49021 

Stock  Clerks,  Sales  Floor 

8 

49023 

Cashiers 

29 

51002 

1st  Line  Supervisors,  Clerical  & Administrative 

65 

53102 

Tellers 

5 

53311 

Insurance  Claims  Clerks 

5 

53905 

Teachers’  Aides  & Assistants,  Clerical 

11  1 

55108 

Secretaries,  Except  Legal  & Medical 

75  I 

55305 

Receptionists  & Information  Clerks 

6 

55338 

Bookkeeping,  Accounting  & Auditing  Clerks 

27 

55347 

General  Office  Clerks 

78 

61005 

Police  & Detective  Supervisors 

14 

63014 

Police  Patrol  Officers 

19 

65008 

Waiters  & Waitresses 

18 

65026 

Cooks,  Restaurant 

6 

65038 

Food  Preparation  Workers 

18 

66008 

Nursing  Aides,  Orderlies  & Attendants 

17 

67005 

Janitors  & Cleaners 

34 

85119 

Other  Machinery  Maintenance  Mechanics 

4 

85132 

Maintenance  Repairers,  General  Utility 

39 

87902 

Earth  Drillers,  Except  Oil  & Gas 

8 

92974 

Packaging  & Filling  Machine  Operators 

13 

97102 

Truck  Drivers,  Heavy  or  Tractor  Trailer 

10 

97111 

Bus  Drivers,  Schools 

10 

Figure  11-3 

Example  Level.  Importance,  and  Frequency  Scales 


8.  Social 
Orientation 


Job  requires  preferring  to  work  with  others  rather  than  alone 
and  being  personally  connected  with  others  on  the  job. 


Level 


HIGH 


What  level  of  this  characteristic  is  needed  to  perform  this  job? 

©T 


LOW 


Requires  a high  degree  of 
participation  and  working  closely 
with  other  organization  members . 


©~ 

@-  - 

®-  - 

®-L 

@)  Not  relevant  at  all  for  performance  on  this  job 


Requires  a moderate  degree  of 
participation  and,  at  times,  working 
closely  with  other  organization 
members. 


Requires  little  participation  with 
other  organization  members ,*  usually 
works  alone. 


Table  11-1 

Descriptive  Statistics  Across  All  Occupations  and  Reliability  Estimates  for  Rated  Differences  Between 
Occupations:  Work  Styles 

Variable 


Level Importance 


Descriptor 

M 

SD 

SEMa 

Ikb 

M 

SD 

SEM 

Ik 

1.  Achievement/Effort 

4.69 

0.88 

.52 

.64 

3.58 

0.45 

.31 

.52 

2.  Persistence 

5.20 

0.77 

.44 

.67 

3.68 

0.47 

.26 

.69 

3.  Initiative 

5.32 

0.70 

.40 

.67 

3.83 

0.41 

.26 

.59 

4.  Energy 

4.97 

0.81 

.53 

.58 

3.56 

0.48 

.33 

.52 

5.  Leadership  Orientation 

4.65 

1.05 

.42 

.84 

3.42 

0.63 

.25 

.84 

6.  Cooperation 

5.98 

0.52 

.33 

.59 

4.16 

0.38 

.25 

.57 

7.  Concern  for  Others 

5.20 

0.85 

.51 

.64 

3.79 

0.49 

.31 

.60 

8.  Social  Orientation 

5.05 

0.89 

.48 

.71 

3.58 

0.53 

.32 

.63 

9.  Self-Control 

5.62 

0.82 

.45 

.70 

4.05 

0.45 

.26 

.66 

10.  Stress  Tolerance 

5.61 

0.54 

.36 

.55 

3.98 

0.38 

.23 

.64 

11.  Adaptability/Flexibility 

5.43 

0.52 

.37 

.50 

3.82 

0.34 

.24 

.51 

12.  Dependability 

6.30 

0.36 

.33 

.15 

4.40 

0.25 

.22 

.26 

13.  Attention  to  Detail 

5.99 

0.48 

.32 

.55 

4.18 

0.34 

.22 

.57 

14.  Integrity 

5.74 

0.73 

.44 

.63 

4.11 

0.47 

.27 

.66 

15.  Independence 

5.25 

0.69 

.49 

.50 

3.77 

0.40 

.29 

.46 

16.  Innovation 

4.54 

1.08 

.52 

.77 

3.25 

0.61 

.32 

.73 

17.  Analytical  Thinking 

4.64 

1.12 

.48 

.82 

3.39 

0.64 

.31 

.78 

Note.  Statistics  are  based  on  35  occupations  with  Work  Styles  questionnaire  responses  from  at  least  four 
incumbents  (mean  number  of  incumbents  = 19.5,  median  =11,  harmonic  mean  = 9.81). 

“This  estimate  of  the  standard  error  of  measurement  was  calculated  as  SEM  = SD*  ^/(l  - rt)  . 

bThis  estimate  of  reliability  was  obtained  by  calculating  the  intraclass  correlation  for  k ratings  across 
occupations:  ICC(l,k)  = 1BMS-WMS1/BMS  (Shrout  & Fleiss,  1979),  where  k is  the  harmonic  mean  of 
the  number  of  ratings  provided  on  each  occupation. 
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Table  11-2 

pliability  of  Rated  Differences  Between  Occupations  Considering  Varying  Numbers  ofRaters:  Work 


Variable 


Descriptor 

Level 

ImDortance 

E,a 

£30  b 

Ii 

J*30 

1.  Achievement/Effort 

16 

85 

10 

77 

2.  Persistence 

17 

86 

18 

87 

3.  Initiative 

17 

86 

13 

81 

4.  Energy 

13 

81 

10 

77 

5.  Leadership  Orientation 

36 

94 

36 

94 

6.  Cooperation 

13 

82 

12 

80 

7.  Concern  for  Others 

15 

85 

13 

82 

8.  Social  Orientation 

20 

88 

15 

84 

9.  Self-control 

20 

88 

17 

86 

10.  Stress  Tolerance 

11 

79 

15 

84 

11.  Adaptability/Flexibility 

09 

75 

10 

U*T 

76 

12.  Dependability 

02 

34 

03 

52 

13.  Attention  to  Detail 

11 

79 

12 

80 

14.  Integrity 

15 

84 

17 

86 

15.  Independence 

09 

75 

08 

72 

16.  Innovation 

26 

91 

21 

89 

17.  Analytical  Thinking 

31 

93 

26 

91 

Np,  Reliability  estimates  are  based  on  35  occupations  with  Work  Styles  questionnaire  responses  from 
at  least  four  incumbents  (mean  number  of  incumbents=19.5,  median=l  1,  harmonic  mean=9  81) 

Decimals  are  omitted.  ' 

Single  rater  estimates  of  reliability  were  obtained  by  calculating  the  intraclass  correlation  for  single 
judge  ratings  across  occupations:  ICC(1,1)  = [BNfS-WMSI/BMS  +(k-l)WMS  (Shrout  & Fleiss,  1979) 
where  k is  the  harmonic  mean  of  the  number  of  ratings  provided  on  each  occupation. 

Estimates  of  reliability  for  30  raters  were  obtained  by  applying  the  Spearman-Brown  correction  formula 
to  the  single  rater  reliability  estimates. 
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Table  11-3 

Reliability  of  Rated  Differences  Between  Occupations  Considering  Various  Recoding  Schemes:  Work 
Styles 


Type  of  Scale  and  Recoding  Scheme  Applied 


Descriptor 

Level 

Imoortance 

la 

lb 

Ic 

la 

lb 

1.  Achievement/Effort 

64 

64 

40 

52 

50 

2.  Persistence 

67 

67 

32 

69 

60 

3.  Initiative 

67 

67 

00 

59 

60 

4.  Energy 

58 

58 

07 

52 

47 

5.  Leadership  Orientation 

84 

84 

33 

84 

83 

6.  Cooperation 

59 

59 

00 

57 

56 

7.  Concern  for  Others 

64 

64 

00 

60 

63 

8.  Social  Orientation 

71 

71 

00 

63 

63 

9.  Self-control 

70 

70 

00 

66 

67 

10.  Stress  Tolerance 

55 

55 

00 

64 

65 

11.  Adaptability/Flexibility 

50 

50 

00 

51 

51 

12.  Dependability 

15 

15 

00 

26 

32 

13.  Attention  to  Detail 

55 

55 

21 

57 

53 

14.  Integrity 

63 

63 

27 

66 

67 

15. Independence 

50 

50 

62 

46 

28 

16.  Innovation 

77 

77 

49 

73 

65 

17.  Analytical  Thinking 

82 

82 

57 

78 

70 

Note.  Reliability  estimates  are  based  on  35  occupations  with  Work  Styles  questionnaire  responses  from 
at  least  four  incumbents  (mean  number  of  incumbents=195,  median=l  1,  harmonic  mean=981). 

Reliability  estimates  stipulated  as  l,  were  calculated  using  the  full  eight  point  scale  for  level  and  retaining 
all  of  the  data  for  the  importance.  Reliability  estimates  stipulated  as  & were  calculated  using  a reduced 
seven  point  scale  for  level  and  excluding  the  data  for  the  importance  scale  where  the  rater  marked  “NR” 
on  the  level  scale.  Reliability  estimates  stipulated  as  & were  calculated  using  a binary  coded  scale  for 
level  (relevant/not  relevant).  Decimals  are  omitted. 


; u 30 


Table  ll-4a 

^nalysis  of  Variance  for  Descriptor,  Occupation,  and  Relevant  Interactions  as  Sources  of  Variation  on 
the  Level  Scale:  Work  Styles 


Source  of  Variation 

SS 

df 

MS 

F 

Occupations 

1908.43 

34 

56.13 

3.15* 

S(Occupations) 

11516.11 

646 

17.83 

Descriptor 

1491.09 

16 

93.19 

66.33* 

Descriptor  x Occupations 

2577.73 

544 

4.74 

3.37* 

Descriptor  x S(Occupations) 

14522.43 

10336 

1.41 

Not^.  Occupations  are  treated  as  random,  between-subjects  effects,  while  descriptors  are  treated  as  fixed 
within-subjects  effects.  ’ 

*P<.04 


Table  ll-4b 

Analysis  of  Variance  for  Descriptor.  Occupation,  and  Relevant  Interactions  as  Sources  of  Variation  on 
the  Importance  Scale:  Work  Styles 


Source  of  Variation 

SS 

df 

MS 

F 

Occupations 

607.86 

34 

17.88 

2.80* 

S(Occupations) 

4118.02 

646 

6.37 

Descriptor 

553.15 

16 

34.57 

60.30* 

Descriptor  x Occupations 

953.31 

544 

1.75 

3.06* 

Descriptor  x S(Occupations) 

5925.73 

10336 

.57 

Note.  Occupations  are  treated  as  random,  between-subjects  effects,  while  descriptors  are  treated  as  fixed, 
within-subjects  effects. 

*E<.05 


Table  11-5 

Interrater  Agreement  Coefficients  for  Each  Scale  Type:  Work  Styles 


Number  of  Raters  on  Each  Variable 

Scale  Type 

Ik 

II 

130 

Level 

70 

19 

88 

Importance 

67 

17 

86 

Note.  Interrater  agreement  coefficient  estimates  are  based  on  35  occupations  with  Work  Styles 
questionnaire  responses  from  at  least  4 incumbents  (mean  number  of  incumbents  = 19.54,  median  =11, 
harmonic  mean  = 9.80).  Full  sample  interrater  agreement  coefficients  (t*  ) were  obtained  by  considering 
the  “Descriptor  x Occupations”  terms  from  Tables  1 l-4a,  1 l-4b,  and  1 l-4c  as  true  variance.  Error 
variance  was  defined  as  the  “Descriptor  x S(Occupations)”  term.  Estimates  of  reliability  for  1 and  30 
raters  were  obtained  by  applying  the  Spearman-Brown  correction  formula  to  the  k rater  reliability 
estimates,  where  k is  the  harmonic  mean  of  the  number  of  raters  for  each  occupation.  Decimals  are 
omitted. 


Table  ll-6a 

Analysis  of  Variance  for  Aggregate  Descriptor.  Occupation,  and  Relevant  Interactions  as  Sources  of 
Variation  on  the  Level  Scale:  Work  Styles 


Source  of  Variation 

SS 

df 

MS 

F 

Occupations 

770.39 

34 

22.66 

3.10* 

S(Occupations) 

4719.71 

646 

7.31 

Aggregate 

464.12 

6 

77.35 

85.10* 

Aggregate  x Occupations 

773.54 

204 

3.79 

4.17* 

Aggregate  x S(Occupations) 

3523.08 

3876 

.91 

Note.  Occupations  are  treated  as  random,  between-subjects  effects,  while  aggregate  descriptors  are 
treated  as  fixed,  within-subjects  effects. 

*P<.05 


Table  ll-6b 

Analysis  of  Variance  for  Aggregate  Descriptor,  Occupation,  and  Relevant  Interactions  as  Sources  of 
Variation  on  the  Importance  Scale:  Work  Styles 


Source  of  Variation 

SS 

df 

MS 

F 

Occupations 

244.71 

34 

7.20 

2.71* 

S(Occupations) 

1715.05 

646 

2.65 

Aggregate 

183.69 

6 

30.61 

82.04* 

Aggregate  x Occupations 

281.95 

204 

1.38 

3.70* 

Aggregate  x S (Occupations) 

1446.39 

.3876 

.37 

Note.  Occupations  are  treated  as  random,  between-subjects  effects,  while  aggregate  descriptors  are 
treated  as  fixed,  within-subjects  effects. 

*P<.05 
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Table  11-7 

Interrater  Agreement  Coefficients  for  Aggregate  Descriptors  for  Each  Scale  Type:  Work  Styles 


Number  of  Raters  on  Each  Variable 

Scale  Type 

Ek 

II 

£30 

Level 

76 

25 

91 

Importance 

73 

22 

89 

Note.  Interrater  agreement  coefficient  estimates  are  based  on  35  occupations  with  Work  Styles 
questionnaire  responses  from  at  least  4 incumbents  (mean  number  of  incumbents  = 19.54,  median  =11, 
harmonic  mean  = 9.81).  Full  sample  interrater  agreement  coefficients  (t* ) were  obtained  by  considering 
the  “Aggregate  x Occupations”  terms  from  Tables  1 l-6a,  1 l-6b,  and  1 l-6c  as  true  variance.  Error 
variance  was  defined  as  the  “Aggregate  x S(Occupations)”  term.  Estimates  of  reliability  for  1 and  30 
raters  were  obtained  by  applying  the  Spearman-Brown  correction  formula  to  the  k rater  reliability 
estimates,  where  k is  the  harmonic  mean  of  the  number  of  raters  for  each  occupation.  Decimals  are 
omitted. 


Table  11-8 

Meains  and  Standard  Deviations  of  Correlations  Between  Level  and  Importance  Scales  Across 
Occupations  and  Descriptors:  Work  Styles 


Level 

Importance 

Scale 

na 

M 

SD 

n 

M 

SD 

Level 

Importance 

17 

.90 

.05 

35 

.93 

.11 

Note.  All  correlations  were  calculated  based  on  the  mean  of  ratings  assigned  by  raters  for  a given 
occupation,  descriptor,  and  scale.  Level-Importance  means  above  the  diagonal  were  calculated  by  taking 
the  level  scale  means  on  a given  occupation  for  all  descriptors,  correlating  them  with  importance  scale 
means,  for  that  occupation,  and  then  averaging  them  with  the  correlations  for  other  occupations.  Level- 
Importance  means  below  the  diagonal  were  calculated  by  taking  the  level  scale  means  on  a given 
descriptor  for  all  occupations,  correlating  them  with  importance  scale  means,  for  that  descriptor,  and 
averaging  them  with  correlations  for  other  descriptors. 

Number  of  correlations  averaged,  not  number  of  observations  on  which  correlations  were  calculated. 


Table  11 -9a 

Intercorrelations  of  Descriptors  for  the  Level  Scale  (Occupation-Level  Data):  Work  Styles 
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Table  ll-10a 

Intercorrelations  of  Descriptors  for  the  Level  Scale  (Incumbent-Level  Data):  Work  Styles 
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Table  ll-10b 

Intercorrelations  of  Descriptors  for  the  Importance  Scale  (Incumbent-Level  Data):  Work  Styles 

Descriptor 1 234  56789  10 

1.  Achievement/Effort 
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Table  11-11 

Principal  Components  Analysis  Pattern  Matrix  for  the  Level  Scale:  Work  Styles 


Factor 


Descriptor 

FI 

F2 

F3 

Communality 

1.  Achievement/Effort 

.85 

.07 

.22 

.78 

2.  Persistence 

.87 

.01 

.19 

.79 

3.  Initiative 

.83 

.19 

-.00 

.73 

4.  Energy 

.69 

.37 

-.15 

.64 

5.  Leadership  Orientation 

.85 

.32 

-.07 

.83 

6.  Cooperation 

-.19 

.84 

.08 

.75 

7.  Concern  for  Others 

.09 

.90 

-.11 

.83 

8.  Social  Orientation 

.05 

.84 

.22 

.76 

9.  Self-control 

.00 

.92 

-.02 

.84 

10.  Stress  Tolerance 

.20 

.75 

.29 

.69 

11.  Adaptability/Flexibility 

.53 

.53 

.56 

.87 

12.  Dependability 

.30 

.58 

.64 

.84 

13.  Attention  to  Detail 

.02 

.07 

.92 

.85 

14.  Integrity 

.37 

.52 

.21 

.46 

15.  Independence 

.64 

-.23 

-.04 

.46 

16.  Innovation 

.86 

.06 

.17 

.78 

17.  Analytical  Thinking 

.86 

-.10 

.31 

.84 

Percent  of  Variance 

35 

29 

12 

Eigenvalue 

7.30  . 

3.96 

1.48 

Note.  N = 35.  The  correlation  matrix  was  based  on  means  calculated  at  the  occupation  level.  FI  = 
Surgency,  Achievement,  High  Activity  Level  Orientation,  F2  = People  Orientation,  and  F3  = 
Detail  Orientation.  These  loadings  are  based  on  an  orthogonal  varimax  rotation. 
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Table  1 l-12b 

Descriptor  Means  and  Standard  Deviations  on  the  Importance  Scale  on  Six  Example  Occupations:  Work  Styles 
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Table  11-13 

Rotated  Correlations  Between  Discriminating  Variables  and  Canonical  Discriminant  Functions  for  Level 
Scale:  Work  Styles 


Functions 


Descriptor 

FI 

F2 

F3 

F4 

IF2 

Tl2 

1.  Achievement/Effort 

.36 

.03 

.03 

.05 

.13 

.13 

2.  Persistence 

.43 

-.01 

.12 

.01 

.19 

.14 

3.  Initiative 

.44 

.00 

.12 

.09 

.22 

.14 

4.  Energy 

.16 

-.05 

.09 

.49 

.28 

.11 

5.  Leadership  Orientation 

.62 

.40 

.03 

.17 

.57 

.13 

6.  Cooperative 

-.12 

.13 

.32 

.08 

.14 

.11 

7.  Caring 

-.03 

.33 

.21 

.19 

.19 

.13 

8.  Social 

.20 

.19 

.61 

.07 

.46 

.15 

9.  Self-control 

.04 

.52 

.26 

.10 

.35 

.15 

10.  Stress  Tolerance 

.16 

.36 

.21 

-.14 

.22 

.10 

1 1 . Adaptability/Flexibility 

.26 

.12 

.23 

-.08 

.14 

.10 

12.  Dependability 

.13 

.09 

.23 

-.07 

.08 

.06 

13.  Attention  to  Detail 

-.02 

-.07 

.16 

-.38 

.18 

.10 

14.  Integrity 

.03 

.48 

-.04 

-.05 

.24 

.13 

15.  Independence 

-.02 

.02 

-.46 

.08 

.22 

.10 

16.  Innovate 

.54 

-.01 

-.02 

.13 

.31 

.19 

17.  Analytical 

.60 

.10 

-.14 

-.19 

.43 

.22 

Re 

.61 

.53 

.44 

.43 

Percent  of  Variance 

26 

18 

11 

10 

Eigenvalues 

.60 

.40 

.24 

.22 

Statistics  are  based  on  35  occupations  with  Work  Styles  questionnaire  responses  from  at  least  4 
incumbents  (mean  number  of  incumbents  = 18.9,  median  = 13,  harmonic  mean  = 9.68).  FI  = 
Achievement/Intelligence;  F2  = Adjustment;  F3  = Interpersonal  Orientation;  F4  = Energy. 

2E  = Sum  of  squared  rotated  standardized  discriminant  function  coefficients  across  four  functions. 
T|2  = Variance  in  Work  Styles  Level  Scale  ratings  accounted  for  by  occupations. 
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Table  11-17 

Correlations  Among  Group  Mean  Ratings  Profiles  and  Within  Three  Example  Occupations:  Work  Styles 


Occ.  1 
Group  1 
(n  = 33) 

Occ.  1 
Group  2 
(n  = 32) 

Occ.  2 
Group  1 
(n  = 38) 

Occ.  2 
Group  2 
(n  = 37) 

Occ.  3 
Group  1 
(n  = 39) 

Occ.  3 
Group  2 
(n  = 39) 

Occ.  1 Group  1 

1.00 

Occ.  1 Group  2 

.94 

1.00 

Occ.  2 Group  1 

.65 

.68 

1.00 

Occ.  2 Group  2 

.58 

.64 

.93 

1.00 

Occ.  3 Group  1 

.56 

.66 

.91 

.91 

1.00 

Occ.  3 Group  2 

.66 

.73 

.95 

.94 

.94 

1.00 

Note.  Within  each  occupation,  raters  were  randomly  grouped  into  halves. 

Occ.  1 = First-line  Supervisors  and  Managers/Supervisors,  Clerical  and  Administrative  Support  Workers, 
Occ.  2 = Secretaries  Except  Legal  and  Medical,  Occ.  3 = General  Office  Clerks. 
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Research  Institutes,  Inc.,  Management  Research  Institute,  Inc.,  Jeanneret  & Associates,  Inc.,  and 
Westat,  Inc.,  performed  the  work  described  herein  under  Contract  Number  94-542,  administered 
by  the  Utah  Department  of  Employment  Security,  on  behalf  of  the  U.S.  Department  of  Labor. 

Under  this  contract,  the  American  Institutes  for  Research  and  its  subcontractors 
developed  an  operational  prototype  for  an  occupational  data  collection,  analysis,  and 
dissemination  system~the  Occupational  Information  Network  or  0*NET~to  replace  the 
Department  of  Labor’s  Dictionary  of  Occupational  Titles.  This  report,  0*NET  Final  Technical 
Report,  submitted  by  the  American  Institutes  for  Research  as  a major  deliverable  under  this 
contract,  describes  the  empirical  evidence  provided  by  the  preliminary  data  collection  effort  for 
the  meaningfulness  of  the  prototype  system.  An  earlier  report.  Development  of  Prototype 
Occupational  Information  Network  (0*NET)  Content  Model,  described  the  development  of  the 
model  underlying  the  0*NET  and  the  design  of  the  questionnaires  used  to  collect  the 
occupational  information.  A separate  report,  0*NET:  An  Information  System  for  the  Workplace. 
Designing  an  Electronic  Infrastructure  (Rose,  Hesse,  Silver,  & Dumas,  1996),  describes  the 
development  of  the  electronic  database  and  provides  technical  documentation  for  the  database. 

The  Holland  Occupational  Codes  and  explanatory  text  included  in  Chapter  10  of  this 
document  are  adapted  and  reproduced  by  special  permission  of  the  Publisher,  Psychological 
Assessment  Resources,  Inc.,  Odessa,  FL  33556,  from  the  Dictionary  of  Holland  Occupational 
Codes,  Second  Edition  by  Gary  D.  Gottfredson,  Ph.D.,  and  John  L.  Holland,  Ph.D.,  Copyright, 
1982,  1989.  Further  reproduction  for  any  purpose  or  by  any  means  is  prohibited  without  the  prior 
written  permission  of  the  Publisher. 

Please  note  that  the  analysis  results  tables  in  Chapters  3 through  11  are  numbered 
uniformly  across  chapters.  Because  some  analyses  are  not  appropriate  for  every  domain, 
some  domains  are  missing  certain  table  numbers.  This  is  intentional.  Please  refer  to 
chapter  2 and,  particularly,  Figure  2-17  for  a listing  of  what  tables  should  appear  in  each 
chapter. 
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The  0*NET  occupational  information  system  relies  on  empirical  data— ratings  provided 
by  job  incumbents  and  occupational  analysts— as  the  bases  for  its  descriptions  of  occupations. 
This  is  one  of  its  strengths,  as  demonstrated  throughout  this  report.  However,  the  0*NET’s 
reliance  on  data  collected  from  job  incumbents  and  occupational  analysts— indeed  the  reliance  of 
most  job  analysis  efforts  to  similar  ratings— may  also  be  considered  problematic  by  those  who 
question  the  accuracy  of  such  ratings.  Morgeson  and  Campion  (1996),  in  particular,  have  recently 
highlighted  concerns  as  to  whether  job  incumbents  and  occupational  analysts  can  and  do  provide 
accurate  ratings  of  occupations  in  job  analysis  studies.  In  their  article,  they  have  described  a 
myriad  of  social  and  cognitive  factors  that  might  influence  the  accuracy  of  ratings  and  of  the 
resulting  occupation  descriptions. 
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There  are  two  ways  in  which  the  0*NET  occupational  information  system  might  be  used 
to  address  concerns  about  the  accuracy  of  job  analysis  data.  At  the  particular  level,  we  might 
investigate  whether  the  0*NET  data  are  likely  to  be  accurate,  and  explore  the  implications  of  the 
findings  for  the  usefulness  of  the  0*NET  system.  Alternately,  we  could  use  the  0*NET  data  as 
an  exemplar  data  set  in  an  investigation  of  general  issues  in  job  analysis  rating  accuracy.  The 
0*NET  already  has  a large  number  of  ratings  over  a variety  of  jobs  and  will  continue  to  grow  in 
the  number  Of  occupations  covered  and  the  number  of  ratings  per  occupation.  It  may  be  that  some 
of  Morgeson  and  Campion’s  (1996)  concerns  can  eventually  be  addressed  using  data  from  the 


It  is  important  to  note,  before  we  begin  our  discussion  of  possible  sources  of  variance  in 
job  analysis  ratings,  particularly  as  they  relate  to  the  0*NET,  that  the  0*NET  data  collection 
strategy  was  not  designed  to  support  an  investigation  of  rating  accuracy:  While  it  may  be 
possible,  though  difficult,  to  imagine  designing  and  conducting  a series  of  studies  of  sufficient 
breadth  and  control  to  thoroughly  investigate  a few  of  the  possible  factors  that  might  influence 
rating  accuracy,  given  the  reality  of  limited  funds  and  time,  such  studies  become  very 
improbable,  as  well  as  impractical. 

The  0*NET  data  collection  was  designed  to  inexpensively  and  efficiently  account  for 
many  of  the  potential  sources  of  variance  by  employing  random  sampling  at  several  stages  of 
rater  selection.  As  detailed  in  Chapter  2,  establishments  were  selected  at  random  from  specific 
Standard  Industrial  Classification  (SIC)  divisions,  and  raters  were  selected  at  random  within 
occupations  within  the  organization.  Because  of  the  scope  of  the  data  collection,  it  was  (and  will 
continue  to  be)  possible  to  include  a wide  variety  of  raters  with  a variety  of  personal 
characteristics  and  from  a variety  of  work  environments.  However,  although  the  data  collection 


0*NET. 
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design  precludes  careful  examination  of  many  of  the  potential  sources  of  error,  the  coherence  and 
sensibility  of  the  results  of  the  analyses  presented  throughout  this  report— particularly  the  high 
interrater  agreement  coefficients  and  the  similarity  of  the  job  incumbents’  and  occupational 
analysts’  ratings— suggest  that  the  ratings  largely  reflect  “true”  variance. 

Bearing  in  mind  that  the  0*NET  was  not  designed  to  support  such  an  investigation,  we 
will  nonetheless  examine  the  characteristics  of  the  already-collected  data,  and  suggest  what 
implications  potential  findings  might  have  on  our  understanding  of  sources  of  variance  in 
general,  and  specifically  as  they  relate  to  the  0*NET.  First,  we  will  describe  a model. 

Model 

The  potential  sources  of  variance  discussed  in  this  chapter  include  a variety  of  factors, 
such  as  raters’  levels  of  familiarity  with  the  occupation,  organizations’  attributes,  questionnaire 
features,  and  a number  of  other  factors  that  may  influence  occupational  ratings.  It  is  important  to 
bear  in  mind  that,  while  some  of  these  may  be  construed  as  sources  of  error,  others  are  generally 
considered  true  variance.  As  Morgeson  and  Campion  (1996)  point  out,  some  of  the  sources  of 
variance,  including  uniformity  of  data  collection  media,  may  actually  inflate  the  observed 
reliability  of  the  ratings  by  subtly  biasing  respondents  to  provide  particular  ratings. 

The  potential  sources  of  variance  in  occupational  ratings  that  we  will  consider  here  are 
outlined  in  Figure  12-1.  This  figure  illustrates  our  model,  explaining  the  way  in  which  different 
factors  may  contribute  to  inaccuracies  in  final  occupational  ratings.  The  examples  used  here 
apply  to  the  0*NET,  but  most  are  relevant  for  other  job  analysis  efforts. 

To  the  far  left  of  the  figure  is  a box  representing  the  actual  occupational  demands.  These 
demands  are  the  reality  we  would  like  to  reflect  in  the  0*NET  occupational  information  system. 
However,  we  recognize  that  the  breadth  of  our  occupational  analysis  unit— for  example,  whether 
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we  are  targeting  the  fairly  general  category  of  mechanic  or  the  more  specific  subspecialty  of 
aircraft  mechanic- will  have  some  impact  on  how  the  occupation’s  features  will  be  summarized, 
and  on  what  activities  and  requirements  will  be  considered  typical.  We  also  recognize  that 
occupations  may  differ  from  establishment  to  establishment,  within  a given  occupation 
depending  on  such  factors  as  the  industry  type  (for  example,  an  electrical  engineer  for  an 
electronics  manufacturing  firm  might  have  different  responsibilities  from  an  electrical  engineer 
for  a broadcasting  station),  the  organizational  culture  (for  example,  the  management  structure 
and  degree  of  bureaucratization  in  a company),  the  size  of  the  establishment,  and  the  geographic 
region.  All  of  these  factors  may  influence  an  occupation’s  actual  demands.  The  extent  to  which 
these  factors  vary  within  an  occupation  in  a sample  will  affect  the  occupational  profile  that 
results. 

Clearly,  the  intention  is  for  the  rater  to  appraise  the  occupation  based  solely  on  the  actual 
demands  of  the  occupation,  either  in  a general  sense,  as  in  the  case  of  an  occupational  analyst 
making  the  ratings,  or  in  regard  to  a particular  instance  of  the  occupation,  in  the  case  of  a job 
incumbent.  However,  the  rater’s  appraisal  of  the  occupation  may  be  influenced  by  other  factors. 
These  factors  include  several  attributes  and  experiences  of  the  rater:  personal  attributes  (e.g.,  sex, 
ethnicity,  education);  the  rater’s  familiarity  with  the  occupation  (e.g.,  the  rater’s  status  as 
incumbent  or  occupational  analyst);  the  rater’s  job  level  (e.g.,  incumbent  or  supervisor);  and  the 
rater’s  tenure  on  the  job. 

Finally,  even  assuming  that  the  particular  instance  of  the  occupation  is  not  unusual  and 
the  rater  appraises  the  occupation  accurately,  there  are  still  opportunities  for  added  variance  to 
creep  into  the  ratings.  This  is  reflected  on  the  right  side  of  Figure  12-1.  For  example,  the  reading 
level  on  the  questionnaires  might  be  too  high  for  some  job  incumbents  or  the  terminology  may  be 
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unfamiliar.  The  questionnaire  scales  may  lack  the  categories  needed  to  accurately  describe  a 
particular  occupation.  Incumbent  raters  may  be  motivated  to  provide  inaccurately  favorable 
ratings  for  their  occupation  by  a consciousness  of  the  social  desirability  of  their  choices.  When 
incumbents  generate  ratings  in  focus  groups,  groups  processes-for  example,  motivational  losses, 
pressures  to  conform  to  the  group  norm— may  impact  the  variance  observed  in  ratings  (Morgeson 
& Campion,  1996).  Finally,  the  data  collection  media  (for  example,  paper  and  pencil 
administration  versus  computer  administration)  may  affect  the  ratings  obtained. 

Evidence 

For  some  of  the  sources  of  variance  included  in  the  model  in  Figure  12-1,  the  data  we 
have  collected  are  suggestive  of  their  possible  effects,  even  though  the  0*NET  data  collection 
was  not  designed  with  the  intention  of  enabling  investigation  of  these  factors.  However,  for  other 
factors  included  in  the  model,  we  simply  do  not  have— and  may  never  have— sufficient  data  to 
carry  out  analyses.  For  still  other  factors,  the  nature  of  the  factor  precludes  empirical  testing.  All 
the  factors  in  the  model  will  be  discussed  below. 

Influences  on  Actual  Job  Demands 

Following  our  model,  we  will  first  consider  the  factors-industry  type,  organizational 
culture,  establishment  size,  geographical  region,  and  occupational  unit— that  are  likely  to  affect 
actual  job  demands. 

Industry  Type 

The  establishments  participating  in  the  0*NET  prototype  data  collection  were  selected 
from  establishments  in  Standard  Industrial  Classification  (SIC)  categories  that  were  identified  as 
likely  to  employ  individuals  in  the  80  occupations  targeted  in  the  initial  data  collection.  The 
selection  of  establishments  was  stratified  across  143  selected  SICs  and  across  four  establishment 
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sizes.  Figure  12-2  presents  the  numbers  of  establishments  in  each  SIC  that  contributed  data  to  the 
initial  0*NET  data  collection  database.  (Additional  establishments  also  contributed  data,  but  for 
occupations  with  too  few  respondents  to  be  included  in  these  analyses.)  In  most  occupations, 
incumbents  were  sampled  from  a number  of  establishments  in  a variety  of  SICs.  For  example,  for 
the  occupation  Cashiers,  questionnaires  were  completed  by  78  job  incumbents  from  24 
establishments  in  5 of  the  9 Standard  Industrial  Classification  divisions,  in  this  case, 
Manufacturing;  Transportation,  Communication,  Electric,  Gas  and  Sanitary  Services;  Retail 
Trade;  Services;  and  Public  Administration. 

Eventually,  after  a large  amount  of  data  have  been  collected,  it  may  be  possible  to 
perform  comparisons  of  occupational  ratings  for  individual  occupations  across  industries.  Of 
course,  occupations  will  vary  considerably  with  respect  to  their  occurrence  across  industries.  In 
this  analysis,  industry  could  be  analyzed  at  the  level  of  the  nine  major  SIC  groupings  into  which 
the  establishments  are  classified.  Sufficient  data  are  not  yet  available  to  allow  this  analysis  to  be 
performed  across  occupations.  An  illustrative  analysis  was  performed  for  the  occupation  of 
Secretary,  Except  Legal  and  Medical  and  is  reported  in  Table  12-1.  In  this  analysis,  the  Skills 
Questionnaire  level  ratings  for  12  incumbents  from  four  establishments  in  the  Manufacturing 
sector  are  compared  with  those  for  12  incumbents  from  five  establishments  in  the  Public 
Administration  sector.  In  this  analysis,  no  statistically  significant  differences  between  the  mean 
ratings  for  incumbents  in  the  two  sectors  were  found.  In  other  words,  the  skills  required  for 
Secretaries  in  the  Manufacturing  sector  are  very  similar  to  those  required  for  Secretaries  in  the 
Public  Administration  sector.  However,  for  other  occupations,  in  other  sectors,  differences  in 
ratings  might  well  occur.  In  particular,  Organizational  Context  and  Generalized  Work  Activities 
ratings  might  be  expected  to  differ  across  industry  types. 
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Organizational  Culture 

' By  organizational  culture,  we  mean  such  aspects  of  an  organization  as  its  management 
structure,  the  degree  of  bureaucratization  in  a company,  and  the  company’s  emphasis  on 
employee  responsibility.  Also  included  are  the  degree  to  which  an  organization  is  involved  in  the 
production  or  use  of  new  technology  and,  particularly,  whether  an  organization  can  be  classified 
as  "high  performance."  In  Chapter  2 of  this  report,  we  discuss  common  criteria  for  classification 
as  a high  performance  organization:  for  example,  organizational  emphasis  on  the  importance  of 
innovation  and  technology  in  the  workplace. 

Analysis  of  organizations'  responses  to  the  establishment  version  of  the  Organizational 
Context  questionnaire,  reported  in  Chapter  8,  suggest  the  ways  in  which  organizations  differ  in 
terms  of  organizational  culture.  In  Chapter  3,  Tables  3-  17b  and  3- 17c,  we  compared  the  Skills 
level  ratings  of  incumbents  working  in  organizations  that  could  be  classified  as  high  performance 
(based  on  the  responses  of  their  organizational  representatives  to  the  Organizational  Context 
questions  presented  in  Appendix  A)  with  the  ratings  of  incumbents  working  in  more  traditional 
organizations.  Comparisons  were  made  for  two  occupations— General  Office  Clerks  and  First 
Line  Supervisors,  Clerical/ Administrative.  In  this  analysis,  we  found  that  while  General  Office 
Clerks  in  high  performance  organizations  reported  needing  higher  levels  of  a variety  of  skills, 
First  Line  Supervisors  did  not  (see  Chapter  3 for  a complete  description  of  the  analysis).  This 
analysis  illustrates  an  organizational  culture  feature  that  does  impact  at  least  some  of  the  ratings. 
Establishment  Size 

Establishments  of  widely  varying  sizes  have  contributed  and  will  continue  to  contribute 
data  to  the  0*NET’s  database.  Establishments,  which  are  defined  for  the  purposes  of  the  0*NET 
data  collection  as  single  sites  (so  that  an  organization  may  consist  of  multiple  establishments) 
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must  have  at  least  five  employees  in  order  to  participate,  but  may  have  several  thousand. 
Establishments  participating  in  the  initial  0*NET  data  collection  ranged  in  size  from  0 to  5,000 
full-time  employees,  with  a median  of  70.  The  size  of  the  establishment  is  expected  to  have 
direct  effects  on  the  data  collected  from  the  establishment  (for  example,  on  how  many  employees 
are  available  for  the  data  collection  and  in  how  many  occupations).  It  may  also  have  indirect 
effects  (for  example,  the  duties  of  a mechanical  engineer  in  a small  company  may  be  more  varied 
than  the  duties  of  a mechanical  engineer  in  a large,  specialized  engineering  shop). 

The  version  of  the  Organizational  Context  questionnaire  that  was  administered  to 
organizational  representatives  provides  information  about  the  size,  among  other  characteristics, 
of  the  establishment.  These  data  can  be  combined  with  the  incumbent  data  from  those 
establishments  to  compare  occupational  ratings  across  establishment  size.  Unfortunately,  because 
small  establishments  by  definition  have  fewer  employees,  they  can  contribute  fewer  respondents, 
particularly  to  some  occupations,  such  as  Secretary,  that  only  become  prevalent  in  larger 
organizations.  In  addition,  they  may  be  less  willing  to  allow  employees  to  take  time  on  the  job  to 
complete  the  questionnaires,  thus  further  decreasing  participation.  There  are  currently  an 
insufficient  number  of  job  incumbents  from  small  organizations  in  the  data  base  to  permit  even 
an  exemplary  analysis  comparing  incumbent  responses  by  establishment  size.  A special  effort  to 

target  smaller  organizations  would  likely  be  necessary  to  collect  sufficient  data  to  allow  analyses 
of  this  factor. 

Geographical  Region 

The  geographical  region  in  which  an  establishment  is  located  and  where  an  incumbent 
works  may  also  impact  on  occupational  ratings.  For  example,  a lawyer  practicing  in  New  York 
City  may  experience  different  job  demands  than  a lawyer  in  a rural  Midwestern  community. 
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Organizations  that  have  establishments  in  several  geographical  regions  and  are  centrally 
organized  may  also  have  different  attributes  than  organizations  that  are  located  in  a single  region. 

Information  about  the  geographical  region  in  which  an  establishment  is  located  was 
collected  through  the  organizational  context  questionnaires  administered  to  organizational 
representatives.  The  incumbent  data  collected  within  occupations  across  establishments  in  the 
initial  0*NET  data  collection  is  currently  too  sparse  to  permit  meaningful  analyses. 

Occupational  Unit 

The  breadth  or  specificity  of  the  occupational  categories  that  are  being  described  in  a job 
analysis  may  affect  the  accuracy  of  the  data.  For  example,  use  of  broader  occupational  units,  such 
as  “mechanic,”  might  require  the  collection  and  combination  of  ratings  from  incumbents  in  a 
variety  of  occupational  subspecialties,  including  “automobile  mechanic”  and  “aircraft  mechanic.” 
This  might  result  in  some  loss  of  precision  in  the  resulting  occupational  profiles.  On  the  other 
hand,  occupational  analysts  might  find  it  easier  to  provide  ratings  for  these  broader  categories 
than  for  occupation  subspecialties. 

The  0*NET  is  currently  using  a set  of  1,122  occupational  units,  developed  by  the 
Occupational  Analysis  Field  Centers  (OAFCs).  The  occupational  units  represent  a taxonomy  of 
occupations  that  is  intermediate  in  number  of  categories  between  the  Occupational  Employment 
Statistics  (OES)  taxonomy  and  the  Dictionary  of  Occupational  Titles  (DOT;  U.S.  Department  of 
Labor,  1991)  taxonomy.  Some  of  the  OES  categories  are  subdivided  and  many  of  the  DOT 
categories  are  combined. 

One  possible  analysis  that  might  suggest  the  importance  of  this  factor  would  involve 
subdividing  several  of  the  current  occupational  units  and  collecting  ratings  from  occupational 
analysts  on  those  units.  An  additional  analysis  of  interest  would  be  to  compare  ratings  of  more 
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homogeneous  subgroups  of  incumbents  within  a current  occupational  unit,  perhaps  grouped  on 
the  basis  of  their  indications  of  occupational  knowledge  specialties  on  the  Knowledges 
Questionnaire. 

Influences  on  Job  Appraisal 

The  next  group  of  factors  are  related  to  how  the  rater  perceives  the  job.  Included  here  are 
the  rater  s personal  attributes,  such  as  sex,  ethnicity,  and  education;  the  rater’s  familiarity  with 
the  job;  the  rater’s  job  level;  and  the  rater’s  time  on  the  job. 

Raters’  Personal  Attributes 

The  personal  attributes  of  raters~for  example,  sex,  ethnicity,  and  educational 
achievement— are  very  likely  to  differ  across  occupations.  For  example,  despite  efforts  to  hire 
women  in  occupations  traditionally  dominated  by  men  and,  to  some  extent,  men  into  jobs 
traditionally  held  by  women,  many  occupations  are  still  dominated  by  one  sex.  For  example,  of 
the  95  Registered  Nurses  who  provided  ratings  in  the  initial  0*NET  data  collection,  93  were 
women.  Similarly,  of  the  67  Police  Patrol  Officers  who  provided  ratings,  64  were  men. 

Of  the  2,197  incumbents  who  responded  to  at  least  one  questionnaire  in  the  initial 
0*NET  data  collection,  64%  were  female  and  36%  were  male;  76%  selected  White/Not  of 
Hispanic  Origin  for  their  ethnicity,  8%  selected  African  American/Not  of  Hispanic  Origin,  6% 
selected  Hispanic,  5%  selected  Native  American,  and  3%  selected  Asian/Pacific  American  (the 
remaining  2%  marked  Other  or  did  not  respond);  and  95%  had  at  least  a high  school  diploma, 
with  only  8%  of  those  also  having  a Bachelor’s  Degree. 

To  what  extent  do  these  differences  in  personal  attributes  contribute  to  differences  in 
occupational  ratings?  And,  where  individuals  differ  systematically  across,  but  not  within, 
occupations,  how  do  these  differences  in  personal  attributes  contribute  to  perceived  occupational 
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differences?  The  0*NET  data  collection  is  designed  to  randomly  select  respondents  across  these 
rater  attributes,  in  order  to  best  control  for  the  effects  of  variations  on  ratings.  However,  this 
method  of  control  has  the  result  that,  if  we  decide  we  want  to  use  these  same  data  to  determine 
the  differential  effects  of  various  attributes,  we  must  have  enormous  quantities  of  it.  In  no 
occupations  are  there  enough  data  from  the  initial  0*NET  data  collection  to  perform  analyses  to 
determine  to  what  extent  ratings  made  by  incumbents  within  the  occupation  differ  depending  on 
incumbent  attributes.  It  is  also  conceivable  that  occupational  analysts’  ratings  may  be  influenced 
by  rater  attributes—for  example,  male  analysts  might  tend  to  rate  traditionally  female-dominated 
jobs  less  highly  than  would  female  analysts.  An  analysis  of  variance  might  be  performed  to 
address  this  issue,  with  sex,  race,  and  educational  level  as  possible  sources  of  variance  within  and 
across  occupations.  An  analysis  of  the  effects  of  one  attribute  (race,  for  example)  within  an 
occupation,  would  have  to  carefully  covaiy  out  other  possible  sources  of  variance,  such  as 
differences  in  job  tenure  between  minority  and  majority  job  incumbents  in  an  occupation  that 
until  recently  was  predominately  held  by  individuals  of  a particular  ethnicity.  Personal  attributes 
might  be  expected  to  affect  ratings  in  all  the  domains  to  some  degree,  although  perhaps  most 
notably  in  the  Work  Styles  and  Occupational  Values  domains. 

Raters’  Job  Familiarity:  Incumbents’  Versus  Analysts’  Ratings 

In  five  of  the  domains  (Basic  and  Cross-Functional  Skills,  Generalized  Work  Activities 
[GWAs],  Abilities,  Work  Context,  and  Knowledges),  both  job  incumbents  and  occupational 
analysts  provided  ratings  for  some  occupations.  Job  incumbents  were  individuals  with  at  least  six 
months  experience  in  the  occupation  and  included  not  only  individuals  actually  performing  the 
occupation,  but  also  some  supervisors.  Most  of  the  incumbents  completed  more  than  one  of  the 
nine  questionnaires  and  a few  completed  as  many  as  five. 
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Occupational  analysts  were  trained  raters  (employment  specialists  at  the  OAFCs  and 
Industrial/Organizational  Psychology  graduate  students),  who  familiarized  themselves  with  a 
particular  occupation  by  studying  a list  of  occupation  tasks,  then  provided  ratings  for  that 
occupation.  Each  task  list  was  based  on  occupation  descriptions  from  the  DOT.  Each  occupation 
was  rated  by  at  least  five  raters  independently,  to  minimize  the  effects  of  rater  error.  During  a 
rating  cycle,  each  rater  rated  a set  of  125  occupational  units  on  one  0*NET  Content  Model 
category  for  level,  importance,  and  frequency,  when  applicable. 

While  job  incumbents  have  the  advantage  of  more  detailed  knowledge  of  their  particular 
occupation,  occupational  analysts  might  be  argued  to  have  the  advantage  of  perspective-having 
knowledge  of  a large  number  of  occupations  and  being  able  to  compare  the  particular  occupation 
being  rated  to  the  characteristics  of  other  occupations. 

For  each  of  the  five  domains  in  which  both  analysts  and  incumbents  provided  ratings, 
analyses  comparing  these  ratings  were  performed.  The  results  of  these  analyses  are  reported  in 
each  of  the  domain  chapters.  In  general,  these  analyses  found  the  incumbents’  and  analysts’ 
ratings  to  be  moderately  to  highly  correlated.  In  the  Skills  domain,  for  example,  the  median 
correlation  between  incumbents’  and  analysts’  ratings  was  .73. 

Despite  these  similarities,  there  were  some  differences  between  the  two  sets  of  ratings. 
Again  taking  the  Skills  domain  as  an  example,  analysts’  ratings  for  the  level  of  Skill  required  for 
an  occupation  tended  to  be  lower  than  incumbents’  ratings,  with  the  exception  of  some  of  the 
technical  skills,  which  the  analysts  rated  slightly  higher  than  did  the  incumbents.  Smaller,  but 
consistent,  differences  were  also  found  for  the  importance  ratings. 

Incumbents’  and  analysts’  ratings  were  each  highly  reliable,  across  occupations  and 
across  domains.  Figure  12-3  shows  the  average  interrater  agreement  for  incumbents’  ratings  for 
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the  actual  number  of  raters  per  occupation  in  these  data  (&)  and  the  predicted  agreement  for  30 
raters  per  occupation  (rjo). 

For  analysts’  ratings,  descriptor  category  reliabilities  were  determined  by  computing  the 
agreement  in  ratings  of  five  raters.  The  reliability  of  a single  rater  or  ri  was  estimated  based  on  a. 
sample  of  five  raters  and  statistically  “stepped  up”  as  an  estimate  of  the  reliability  of  five  raters 

t 

(is).  Reliabilities  for  each  descriptor  category  were  then  averaged  across  all  10  cycles  of  ratings, 
or  1 122  Occupational  Units.  Abilities  and  Work  Context  reliabilities  are  based  on  130 
Occupational  Units  pending  completion  of  the  final  rating  cycle.  Values  for  the  reliability  of 
ratings  in  the  descriptor  categories  range  from  .81  for  Generalized  Work  Activities  frequency  to 
.88  for  Generalized  Work  Activities  level,  as  can  be  seen  in  Figure  12-4.  Analyses  of  the  ratings 
confirmed  that  the  descriptors  could  be  reliably  rated  using  the  DOT  task  data. 

In  general,  incumbents  and  analysts  were  found  to  separately  provide  reliable 
Occupational  ratings,  and  their  ratings  are  moderately  to  highly  correlated.  However,  the 
amplitude  of  the  ratings  tended  to  be  smaller  for  the  analysts,  though  this  was  not  true  across  all 
descriptors  within  the  various  domains.  This  finding  raises  interesting  questions  about  the 
absolute  versus  relative  nature  of  the  scaling  and  the  effect  of  occupational  familiarity  on  the 
ratings.  This  finding  guided  our  decision  to  report  analysts’  and  incumbents’  ratings  separately  in 
the  initial  version  of  the  0*NET  database. 

Raters’  Job  Level 

Occupational  ratings  were  provided  both  by  job  incumbents  and  by  their  supervisors 
(both  were  referred  to  as  incumbents  throughout  this  report).  We  might  expect  job  incumbents  to 
be  at  least  as  familiar,  and  possibly  more  familiar,  with  the  particular  requirements  of  their  job  as 
are  supervisors,  but  supervisors  may  be  more  aware  of  variations  in  job  requirements  across  job 
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assignments  and  how  job  requirements  compare  with  the  requirements  of  other  jobs.  We  would 
expect  the  effect  of  job  level  on  ratings  to  extend  across  the  domains. 

Of  2,197  incumbents  in  29  occupations  providing  ratings,  70%  identified  themselves  as 
job  incumbents,  7%  as  supervisors,  and  20%  said  they  both  worked  in  the  job  and  supervised 
others  in  the  job  (3%  did  not  respond  to  the  question).  The  numbers  of  supervisors  may  be 
inflated  because  many  of  the  individuals  in  management-related  occupations  such  as  First  Line 
Supervisor  and  General  Manager  reported  that  they  were  supervisors  of  job  incumbents  in  those 
positions.  It  is  possible  that  they  misunderstood  the  question,  which  requested  that  the  rater  select 
the  category  that  best  describes  your  relationship  to  this  job.”  Without  specifically  targeting 
supervisors,  its  is  unlikely  that  the  0*NET  data  collection  will  result  in  sufficient  numbers  of 
supervisors  to  allow  meaningful  comparisons  of  their  ratings  with  incumbents’  ratings. 

Raters’  Job  Tenure 

The  amount  of  time  a rater  has  spent  in  a job  might  also  be  expected  to  influence 
judgments  made  about  the  job.  In  collecting  these  data,  we  requested  that  job  incumbents  have  at 
least  six  months  of  experience  on  the  job.  In  fact,  the  data  we  collected  in  the  initial  0*NET  data 
collection  were  provided  by  incumbents  with  amounts  of  job  experience  ranging  from  less  than  a 
month  to  more  than  ten  years  (median  job  experience  was  between  six  and  ten  years).  However, 
individuals  with  extensive  experience  in  a single  job  may  lack  experience  in  other  jobs; 
experience  in  other  jobs  might  provide  a perspective  for  rating  the  requirements  of  the  current 
job.  The  interesting  comparison  in  this  case  may  be  between  incumbents  with  very  little  job 
experience  for  example,  less  than  a year—and  those  with  substantial  experience — for  example, 
six  or  more  years.  An  illustration  of  such  an  analysis  is  presented  in  Table  12-2.  In  this  analysis, 
the  Skills  level  ratings  for  14  Secretaries,  Not  Medical  and  Legal  with  less  than  a year  experience 


Occupational  Descriptor  Covariates  12-15 


were  compared  with  those  for  38  Secretaries  with  six  or  more  years  experience.  The  results 
strongly  support  the  notion  that  experience  has  a relatively  minor  effect  on  these  ratings*;  the 
mean  ratings  correlate  .93  and  only  two  of  the  46  descriptors  have  statistically  significant 
differences  at  the  .05  level,  about  what  would  be  expected  by  chance. 

Influences  on  Job  Ratines 

The  last  group  of  factors  that  may  contribute  to  rating  variance  contains  those  factors  that 
influence  not  the  job  itself  or  the  rater’s  perception  of  the  job,  but  only  the  ratings  produced. 
These  factors  are  reading  level,  measurement  scales,  social  desirability,  group  processes,  and  data 
collection  media. 

Reading  Level 

In  creating  the  0*NET  questionnaires,  every  effort  was  made  to  choose  easily  understood 
terms  and  write  the  descriptor  definitions  and  anchors  at  no  more  than  an  eighth  grade  reading 
level.  It  is  probable,  however,  that,  as  with  many  other  job  analysis  questionnaires,  some  portions 
of  the  questionnaires  may  pose  reading  difficulties  for  individuals  in  some  jobs,  especially  those 
that  do  not  have  even  modest  requirements  for  reading  comprehension. 

In  the  try-out  of  the  draft  questionnaires  in  the  Fall  of  1994  (Mumford  & Sager,  1995), 
202  job  incumbents  from  a variety  of  jobs,  ranging  from  typist  to  van  driver  to  occupational 
analyst  completed  drafts  of  the  0*NET  questionnaires.  After  completing  each  questionnaire, 
these  job  incumbents  were  asked:  (1)  Did  you  find  the  instructions  in  this  section  of  the 
questionnaire  easy  to  understand?  and  (2)  Did  you  find  the  questions  easy  to  understand?  On  a 
scale  of  1 to  5,  where  5 is  the  most  positive  and  1 the  most  negative,  average  ratings  ranged  from 
3.67  for  the  instructions  and  3.82  for  the  questions  for  the  Training,  Education,  Licensure,  and 
Experience  Questionnaire  to  4.42  and  4.28  for  the  Occupational  Values  Questionnaire.  Based  on 
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these  results,  the  questionnaire  instructions  were  revised  and  simplified.  While  these 
understandability  ratings  were  quite  high,  we  would  expect  ratings  of  the  current  questionnaires, 
which  use  the  revised  instructions,  to  be  higher. 

This  does  not  mean  that  some  of  the  job  incumbents  asked  to  participate  in  the  initial 
0*NET  data  collection  may  not  have  been  discouraged  from  participating  or  influenced  in  their 

O 

responses  by  the  reading  difficulty  of  the  questionnaires.  In  fact,  some  organizational 
representatives  reported  that  they  were  refusing  to  continue  their  establishment's  participation  in 
the  initial  0*NET  data  collection  because  the  questionnaires  appeared  to  be  too  complex  for  their 
employees  to  understand.  As  far  as  we  can  determine,  this  was  the  organizational  representative’s 
opinion,  and  not  a report  of  problems  encountered  after  distribution. 

Measurement  Scales 

Attributes  of  the  questionnaires  used  to  collect  the  occupational  ratings  can,  of  course, 
affect  the  quality  of  the  data  collected.  For  example,  if  performance  of  a job's  duties  involves  a 
task  that  is  not  included  on  the  occupation-specific  task  questionnaire  or  within  one  of  the 
categories  of  activities  included  on  the  Generalized  Work  Activities  (GWAs)  questionnaire,  no 
data  about  that  task  will  be  collected.  In  creating  the  questionnaires,  the  need  for  completeness 
was  balanced  with  the  equally  important  needs  for  parsimony  and  practicality  in  the  time 
required  for  data  collection.  In  Chapter  16,  which  describes  analyses  of  the  occupation-specific 
questionnaire  data,  for  example,  we  found  that  some  tasks  that  appeared  on  the  occupation- 
specific  questionnaires-certain  law  enforcement  activities,  for  example-could  not  be  readily 
classified  into  any  activity  from  the  GWAs  questionnaire. 

An  additional  feature  of  the  questionnaires  that  directly  impacts  the  quality  of  the  data  is 
the  actual  rating  scales  on  which  job  incumbents  or  occupational  analysts  were  asked  to  make 
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their  ratings.  For  example,  on  several  of  the  questionnaires  (i.e..  Skills,  Abilities,  Knowledges, 
Work  Styles,  and  GWAs),  anchors  were  provided  for  the  level  rating  scale.  Raters  were  asked  to 
make  their  ratings  in  relation  to  these  examples  or  descriptions  of  low,  medium,  and  high  levels 
of  the  descriptor.  Although  these  anchors  were  provided  to  aid  raters  in  understanding  the 
descriptor  and  placing  their  ratings  on  a common  scale  across  occupations,  it  is  possible  that  the 
particular  examples  given  may  have  influenced  the  ratings  in  ways  not  intended.  For  example, 
one  of  the  anchors  for  the  Skills  questionnaire  descriptor,  Reading  Comprehension,  is  “Reading  a 
memo  from  management  describing  new  personnel  policies.”  It  is  possible  that  an  individual 
rater's  associations  or  experience  with  this  activity  may  adversely  affect  the  accuracy  of  his  or  her 
ratings  for  this  descriptor— for  example,  a recent  bad  experience  with  organizational  personnel 
representatives.  It  is  difficult,  however,  for  us  to  conjure  up  examples  that  plausibly  could  have 
serious,  systematic  effects. 

Social  Desirability 

Social  psychologists  have  suggested  that  respondents  may  tend  to  respond  to  questions  in 
ways  that  they  think  will  place  them  in  a positive  light  (e.g.,  Crowne  & Marlowe,  1964).  To 
reduce  the  impact  of  this  source  of  variance,  the  0*NET  data  in  the  initial  data  collection  were 
collected  under  conditions  of  anonymity;  job  incumbents  could  mail  their  questionnaires  back  or 
return  them,  sealed,  to  coordinators  within  their  establishments.  Even  so,  they  may  have 
experienced  some  anxiety  about  their  ratings.  Incumbents  might  be  expected  to  be  most  likely  to 
attempt  to  give  socially  desirable  responses  in  such  domains  as  Knowledges  and  Work  Styles, 
but  such  effects  could  occur  in  almost  any  domain. 

The  effect  of  social  desirability  is  difficult  to  assess,  because  it  may  tend  to  inflate  ratings 
uniformly  across  incumbent  raters.  One  would  expect  occupational  analysts,  who  are  presumably 
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rating  an  occupation  objectively,  to  not  be  influenced  by  this  factor.  However,  occupational 
analysts  and  job  incumbents’  ratings  may  differ  for  other  reasons  having  to  do  with  actual  or 
“true”  occupational  factors,  as  well,  making  it  difficult  to  determine  what  part  of  the  difference 
might  be  due  to  the  effect  of  social  desirability  pressures  on  job  incumbents. 

Group  Processes 

The  initial  0*NET  data  were  collected  from  job  incumbents  and  occupational  analysts, 
who  completed  the  domain  questionnaires  individually.  Therefore,  group  processes,  such  as 
careless  responding  or  being  influenced  by  the  opinions  held  by  others  in  the  group  are  likely  to 
be  more  relevant  in  future  data  collections,  which  may  include  collection  of  data  through  focus 
groups.  However,  the  tendency  to  respond  carelessly  or  thoughtlessly  may  have  had  an  impact  in 
this  data  collection  as  well,  particularly  if  incumbents  perceived  themselves  as  being  a part  of  a 
larger  group  that  was  providing  ratings.  Promises  of  anonymity  may  have  heightened  such  an 
effect.  One  analysis  that  might  shed  light  on  the  extent  to  which  incumbents  responded 
thoughtfully  in  making  ratings  is  an  analysis  of  the  not  relevant  responses,  because  not  relevant 
responses  might  be  seen  as  an  easy  response  if  one  were  interested  in  hurrying  through  the  rating 
process.  Comparisons  of  the  reliabilities  of  the  full-scale  ratings  with  the  reliabilities  of  the  same 
ratings  dichotomized  into  relevant/not  relevant  (for  example,  those  reported  in  Chapter  16  for  the 
occupation-specific  task  ratings)  suggest  that  respondents  are  making  the  relevant/not  relevant 
distinction  reliably  within  occupation  and  descriptor.  Additionally,  we  expected  that  rater  set 
might  lead  to  random  responding.  However,  the  reliabilities  of  the  full-scale  ratings  tend  to  be 
high  (for  example,  for  the  Skills  level  scale,  they  range  from  .75  to  .92)  across  almost  all 
questionnaires  and  scales,  suggesting  that  raters  are  attending  to  the  rating  task  and  that  random 
responding  is  not  a problem. 
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Data  Collection  Media 

All  the  data  analyzed  in  preparing  this  report  were  collected  using  paper-and-pencil 
versions  of  the  0*NET  questionnaires.  The  versions  of  the  questionnaires  used  by  incumbents  to 
provide  ratings  and  those  used  by  analysts  differ  slightly-for  example,  analysts  do  not  provide 
ratings  for  whether  particular  skills  are  required  at  job  entry.  However,  the  differences  are  minor. 
As  Morgeson  and  Campion  (1996)  point  out,  using  the  same  medium  to  collect  occupational 
information  across  raters  may  have  the  paradoxical  effect  of  artificially  inflating  the  reliability  of 
the  ratings.  Until  comparable  data  are  available  from  other  media,  it  will  be  impossible  to  assess 
the  amount  of  variance  in  occupational  ratings  that  is  due  to  the  paper-and-pencil  form  of  the 
measures.  Electronic  versions  of  the  0*NET  questionnaires  have  been  prepared  and  will  provide 
some  data  for  such  a comparison.  The  electronic  versions  are  very  similar  in  appearance  and 
identical  in  wording  and  structure  to  the  paper-and-pencil  versions,  however.  A more  convincing, 
but  much  less  practical,  demonstration  might  be  comparison  of  the  questionnaire  results  with 
information  about  the  same  occupation  and  worker  attributes  based  on  job  observations,  critical 
incidents  analysis,  or  focus  groups.  In  such  a comparison,  however,  we  would  introduce  other 
complicating  factors,  such  as  whether  the  definitions  of  the  descriptors  were  adequately  and 
comparably  conveyed. 

Discussion  and  Conclusions 

The  0*NET  occupational  information  system  is  undoubtedly  one  of  the  largest 
occupational  analysis  efforts  ever  undertaken,  and,  as  such,  could  afford  a valuable  opportunity 
to  investigate  data  accuracy  issues  that  are  of  concern  in  many  job  analyses.  Additionally,  its 
reliance  on  empirical  data  heighten  concerns  about  the  accuracy  of  the  0*NET  occupation 
descriptions  based  on  those  data. 
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The  analysis  results  presented  elsewhere  in  this  volume  support  the  usefulness  of  the 
0*NET  data.  Although  the  0*NET  data  do  not  lend  themselves  to  analyses  of  most  of  the 
potential  sources  of  rating  variance,  the  evidence  is  strong  that  the  ratings  are  accurately 
describing  the  targeted  jobs— for  example,  agreement  among  raters  within  job  is  high,  even  across 
incumbents  and  analysts.  Additionally,  the  results  for  particular  occupations  are  consistent  with 
our  prior  knowledge  of  the  jobs. 

Because  the  0*NET  data  collection  was  designed  to  control  many  of  the  potential  sources 
of  rating  variance  through  random  sampling,  and  was  not  designed  to  measure  the  effects  of  these 
factors,  it  may  not  be  especially  well-suited  for  use  in  investigations  of  potential  sources  of  rating 
variance.  However,  it  is  our  hope  that  tentative  results  based  on  the  0*NET  data  may  provide 
some  guidance  for  prioritizing  the  factors  to  be  targeted  in  future  studies. 
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Figure  12-1 

Potential  Sources  of  Variance  in  Q*NET  Ratings 
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Figure  12-2 

Numbers  of  Participating  Establishments  bv  Standard  Industrial  Classification  Division 


Standard  Industrial  Classification  Division 

Number  of 
Establishments 

Agriculture,  Forestry  and  Fishing 

3 

Construction 

2 

Manufacturing 

22 

Transportation,  Communication,  Electric,  Gas  and  Sanitary  Services 

5 

Wholesale  Trade 

5 

Retail  Trade 

15 

Finance,  Insurance  and  Real  Estate 

17 

Services 

98 

Public  Administration 

10 

Figure  12-3 

Estimated  Reliabilities  of  Incumbents’  Ratings 


Questionnaire 

Scale 

Lk 

Skills 

Level 

.79 

.93 

Importance 

.79 

.93 

Job  Entry  Requirement 

.60 

.83 

Knowledges 

Level 

.86 

.95 

Importance 

.85 

.94 

Training,  Education,  Licensure,  & 
Experience 

Instructional  Program 

.78 

.92 

Educational  Subject  Area 

.74 

.90 

Licensure 

.85 

.95 

Experience 

.79 

.93 

Generalized  Work  Activities 

Level 

.80 

.92 

Importance 

.78 

.92 

Frequency 

.74 

.90 

Work  Context 

.87 

.95 

Organizational  Context 

Across  Occupations 

.64 

.84 

Across  Organizations 

.45 

.79 

Abilities 

Level 

.82- 

.93 

Importance 

.82 

.93 

Occupational  Values 

.60 

.82 

Work  Styles 

Level 

.70 

.88 

Importance 

.67 

.86 

Note:  rk  is  the  observed  interrater  agreement  coefficient;  £30  is  the  estimated  interrater  agreement 
coefficient  for  30  raters. 


Figure  12-4 

Estimated  Reliabilities  of  Analysts’  Ratings 


Questionnaire 

Scale 

Average  is 

Abilities 

Level 

.87 

Importance 

.85 

Generalized  Work  Activities 

Level 

.88 

Importance 

.85 

Frequency 

.81 

Knowledges 

Level 

.83 

Importance 

.83 

Skills 

Level 

.87 

Importance 

.84 

Work  Context 

.82 

Note:  £5  is  the  average  estimated  reliability  of  analyst  ratings  for  five  raters. 


Table  12-1 

Comparison  of  Level  Scale  Ratings  for  Secretaries.  Not  Medical  and  Legal  in  Establishments  in 
the  Manufacturing  and  Public  Administration  Sectors:  Basic  and  Cross-Functional  Skills 


Manufacturing  Public  Administration 


(n  = 

12) 

(n  = 

12) 

Descriptor 

M 

SD 

M 

SD 

t 

F 

1. 

Reading  Comprehension 

4.58 

1.16 

4.83 

1.27 

-0.50 

1.20 

2. 

Active  Listening 

3.67 

1.72 

4.42 

1.31 

-1.20 

1.72 

3. 

Writing 

3.75 

1.06 

4.42 

1.24 

-1.42 

1.37 

4. 

Speaking 

2.67 

2.23 

3.92 

1.40 

-1.53 

2.54 

5. 

Mathematics 

2.08 

1.88 

2.17 

1.34 

-0.13 

1.97 

6. 

Science 

0.08 

0.29 

0.17 

0.58 

-0.45 

4.00* 

7. 

Critical  Thinking 

2.50 

1.88 

2.00 

1.91 

0.65 

1.03 

8. 

Active  Learning 

1.92 

1.98 

2.75 

1.14 

-1.27 

3.02 

9. 

Learning  Strategies 

2.58 

1.88 

2.17 

1.53 

0.60 

1.51 

10. 

Monitoring 

3.00 

0.95 

2.25 

1.82 

1.27 

3.67* 

11. 

Social  Perceptiveness 

3.33 

1.44 

2.17 

1.59 

1.89 

1.22 

12. 

Coordination 

2.42 

2.27 

3.00 

1.76 

-0.70 

1.66 

13. 

Persuasion 

2.75 

1.76 

1.58 

1.56 

1.71 

1.27 

14. 

Negotiation 

1.75 

1.86 

1.83 

1.70 

-0.11 

1.20 

15. 

Instructing 

3.33 

1.78 

3.67 

0.89 

-0.58 

4.00* 

16. 

Service  Orientation 

3.33 

1.78 

2.92 

1.44 

0.52 

1.53 

17. 

Problem  Identification 

3.67 

1.50 

2.58 

1.93 

L54 

1.66 

18. 

Information  Gathering 

2.92 

0.90 

3.42 

1.93 

-0.81 

4.60* 

19. 

Information  Organization 

2.83 

2.12 

2.42 

2.07 

0.49 

1.05 

20. 

Synthesis/  Reorganization 

3.42 

1.16 

2.83 

1.53 

1.05 

1.74 

21. 

Idea  Generation 

1.50 

1.73 

1.82 

1.66 

-0.45 

1.09 

22. 

Idea  Evaluation 

2.50 

1.51 

1.83 

1.47 

1.10 

1.06 

23. 

Implementation  Planning 

1.83 

1.90 

1.75 

1.42 

0.12 

1.79 

24. 

Solution  Appraisal 

2.75 

1.36 

1.75 

1.54 

1.68 

0.28 

25. 

Operations  Analysis 

1.42 

1.24 

1.58 

1.83 

-0.26 

2.18 

26. 

Technology  Design 

0.50 

1.17 

0.00 

0.00 

1.48 

— 

27. 

Equipment  Selection 

2.83 

2.08 

2.42 

1.56 

0.55 

1.78 

28. 

Installation 

0.33 

0.89 

0.17 

0.58 

0.55 

2.35 

29. 

Programming 

0.08 

0.29 

0.00 

0.00 

1.00 

— 

30. 

Testing 

0.00 

0.00 

0.00 

0.00 

— ■ 

— 

31. 

Operation  Monitoring 

0.00 

0.00 

0.33 

0.78 

-1.48 

— 

32. 

Operation  and  Control 

2.00 

0.85 

1.50 

1.45 

1.03 

2.91 

33. 

Product  Inspection 

2.42 

1.38 

2.00 

1.81 

0.63 

1.72 

34. 

Equipment  Maintenance 

1.00 

1.65 

0.92 

1.31 

0.14 

1.59 

35. 

Troubleshooting 

0.67 

0.98 

1.50 

1.68 

-1.48 

2.94 

36. 

Repairing 

0.75 

1.06 

0.75 

1.22 

0.00 

1.32 

37. 

Visioning 

2.08 

1.56 

1.33 

1.50 

1.20 

1.08 

38. 

Systems  Perception 

1.00 

1.13 

1.50 

1.73 

-0.84 

2.34 
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Table  12-1  (continued) 

Comparison  of  Level  Scale  Ratings  for  Secretaries,  Not  Medical  and  Legal  in  Establishments  in 
the  Manufacturing  and  Public  Administration  Sectors:  Basic  and  Cross-Functional  Skills 

Manufacturing  Public  Administration 


Descriptor 

(n=  12) 

(n  = 

12) 

M 

SD 

M 

SD  ! 

t 

F 

39. 

Identification  of  Downstream 
Consequences 

1.50 

1.38 

1.00 

1.81 

0.76 

1.72 

40. 

Identification  of  Key 
Causes 

1.42 

1.98 

2.00 

2.04 

-0.71 

1.06 

41. 

Judgment  and  Decision 
Making 

1.83 

2.17 

2.08 

2.07 

-0.29 

1.10 

42. 

Systems  Evaluation 

1.17 

1.85 

0.55 

1.51 

0.89 

1.50 

43. 

Time  Management 

3.33 

1.37 

3.42 

1.83 

-0.13 

1.78 

44. 

Management  of 
Financial  Resources 

2.08 

2.02 

1.92 

2.15 

0.20 

1.13 

45. 

Management  of 
Material  Resources 

2.08 

1.73 

1.67 

1.72 

0.59 

1.01 

46. 

Management  of 

1.75 

1.60 

1.67 

1.72 

0.12 

1.16 

Personnel  Resources 


Note.  Statistics  are  based  on  Skills  questionnaire  responses  from  12  incumbents  from  four 
establishments  in  the  Manufacturing  sector  and  12  incumbents  from  five  establishments  in  the 
Public  Administration  sector. 

The  t statistic  tests  for  differences  in  group  means. 

The  F statistic  tests  for  differences  in  group  standard  deviations. 

*P  < .05 


Table  12-2 

Comparison  of  Level  Scale  Ratings  for  Secretaries,  Not  Medical  and  Legal  With  Less  Than  a 
Year  and  Six  or  More  Years  Experience:  Basic  and  Cross-Functional  Skills 


Descriptor 

Less  Than  a Year 
(n=14) 

Six  or  More  Years 
(n  = 38) 

M 

SD 

M 

SD 

t 

F 

1.  Reading  Comprehension 

4.71 

1.77 

4.66 

1.07 

0.11 

2.74* 

2;  Active  Listening 

4.64 

1.86 

4.50 

0.98 

0.27 

3.60* 

3.  Writing 

4.71 

0.99 

4.43 

1.17 

0.86 

1.40 

4.  Speaking 

4.29 

1.68 

3.55 

1.62 

1.41 

1.08 

5.  Mathematics 

3.43 

2.10 

2.34 

1.77 

1.72 

1.41 

6.  Science 

1.29 

2.05 

0.32 

1.07 

1.68 

3.67* 

7.  Critical  Thinking 

3.50 

1.99 

3.39 

1.70 

0.18 

1.37 

8.  Active  Learning 

3.43 

2.28 

3.29 

1.75 

0.21 

1.70 

9.  Learning  Strategies 

3.43 

1.45 

3.18 

1.67 

0.52 

1.33 

10.  Monitoring 

2.86 

1.75 

3.29 

1.72 

-0.79 

1.04 

11.  Social  Perceptiveness 

3.93 

1.27 

3.22 

1.55 

1.68 

1.49 

12.  Coordination 

3.07 

1.90 

3.18 

1.74 

-0.19 

1.19 

13.  Persuasion 

3.00 

1.66 

3.08 

1.51 

-0.16 

1.21 

14.  Negotiation 

2.43 

1.65 

2.16 

1.87 

0.51 

1.28 

15.  Instructing 

3.92 

2.18 

3.55 

1.50 

0.57 

2.11 

16.  Service  Orientation 

3.93 

1.90 

3.45 

1.86 

0.82 

1.04 

17.  Problem  Identification 

3.36 

2.13 

3.55 

1.70 

-0.31 

1.57 

18.  Information  Gathering 

4.14 

1.41 

3.66 

1.42 

1.10 

1.01 

19.  Information  Organization 

3.57 

2.17 

3.37 

1.75 

0.31 

1.54 

20.  Synthesis/  Reorganization 

3.36 

1.45 

3.16 

1.10 

0.47 

1.74 

21.  Idea  Generation 

2.43 

1.83 

2.22 

1.70 

0.38 

1.16 

22.  Idea  Evaluation 

2.64 

1.74 

2.66 

1.36 

-0.03 

1.64 

23.  Implementation  Planning 

2.43 

1.79 

2.47 

1.69 

-0.08 

1.12 

24.  Solution  Appraisal 

2.29 

1.98 

2.79 

1.28 

-0.89 

2.39* 

25.  Operations  Analysis 

1.14 

1.51 

2.21 

1.83 

-2.13* 

1.47 

26.  Technology  Design 

1.79 

2.29 

0.55 

1.29 

1.90 

3.15* 

27.  Equipment  Selection 

3.43 

1.91 

2.76 

1.92 

1.11 

1.01 

28.  Installation 

0.79 

1.63 

0.26 

0.79 

1.15 

4.26* 

29.  Programming 

0.64 

1.65 

0.47 

1.08 

0.36 

2.33* 

30.  Testing 

0.64 

1.65 

0.11 

0.51 

1.20 

10.47* 

3 1 . Operation  Monitoring 

0.43 

0.76 

0.45 

0.89 

-0.08 

1.37 

32.  Operation  and  Control 

1.93 

1.38 

1.84 

1.22 

0.21 

1.28 

33.  Product  Inspection 

2.07 

2.20 

2.39 

1.33 

-0.52 

2.74* 

34.  Equipment  Maintenance 

0.71 

1.07 

0.71 

1.25 

0.01 

1.36 

35.  Troubleshooting 

1.71 

1.68 

1.29 

1.49 

0.83 

1.27 

36.  Repairing 

0.50 

0.76 

0.66 

1.12 

-0.56 

2.17 

37.  Visioning 

1.14 

1.23 

2.03 

1.53 

-2.14* 

1.55 

38.  Systems  Perception 

1.64 

1.50 

1.50 

1.61 

0.30 

1.15 

0 
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Table  12-2  (continued) 

Comparison  of  Level  Scale  Ratings  for  Secretaries.  Not  Medical  and  Legal  With  Less  Than  a 
Year  and  Six  or  More  Years  Experience:  Basic  and  Cross-Functional  Skills 


Less  Than  a Year  Six  or  More  Years 

(n=  14) (n  = 38) 


Descriptor 

M 

SD 

M 

SD 

t 

F 

39. 

Identification  of  Downstream 
Consequences 

1.86 

1.70 

1.29 

1.54 

1.09 

1.22 

40. 

Identification  of  Key 
Causes 

2.50 

2.03 

2.26 

2.05 

0.37 

1.02 

41. 

Judgment  and  Decision 
Making 

2.86 

1.61 

2.29 

1.90 

1.07 

1.39 

42. 

Systems  Evaluation 

1.14 

1.23 

0.84 

1.54 

0.74 

1.57 

43. 

Time  Management 

3.64 

1.28 

4.03 

1.65 

-0.88 

1.66 

44. 

Management  of 
Financial  Resources 

2.14 

2.38 

1.53 

1.98 

0.86 

1.44 

45. 

Management  of 
Material  Resources 

2.71 

1.94 

2.05 

1.71 

1.13 

1.29 

46. 

Management  of 

2.21 

2.15 

1.76 

1.67 

0.71 

1.66 

Personnel  Resources 


Note.  Statistics  are  based  on  Skills  questionnaire  responses  from  14  incumbents  from  1 1 
establishments  with  less  than  a year  of  experience  on  the  job  and  38  incumbents  from  23 
establishments  with  six  or  more  years  of  experience. 

The  t statistic  tests  for  differences  in  group  means. 

The  F statistic  tests  for  differences  in  group  standard  deviations. 

*U  < .05 
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Cross-Domain  Analysis  Results 
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Christopher  E.  Sager 
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The  content  model  underlying  the  0*NET  occupational  information  system  specifies  a 
number  of  domains  that  might  be  used  in  describing  jobs  and  occupations.  The  analyses  for 
descriptors  within  each  of  these  0*NET  content  domains,  described  to  this  point,  show  that  data 
collected  using  the  0*NET  descriptors  are  generally  very  reliable.  These  analyses  also  provide  a 
great  deal  of  information  concerning  the  structure  of  each  domain,  but  most  of  these  analyses  are 
necessarily  internal.  The  within-domain  analyses  do,  however,  provide  preliminary  evidence  that 
the  0*NET  descriptors  discriminate  between  occupations  in  a sensible  manner. 

The  present  chapter  describes  further  analyses  to  assess  the  construct  validity  of  the 
measures.  These  analyses  involved  examining  relationships  between  descriptors  from  the  various 
content  domains,  across  occupations,  and  assessing  the  structure  of  these  cross-domain 
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relationships.  Much  of  the  data  concerning  descriptors  from  different  content  domains  was 
actually  collected  from  different  incumbents  in  the  same  occupations,  and  this  provides  for  an 
even  stronger  test  of  cross-domain  relationships.  These  cross-domain  analyses  can  be  viewed  as 
preliminary  evidence  of  external  validity.  They  also  provide  information  concerning  the  overlap 
between  the  various  content  domains. 

Sample 

There  is  a great  deal  of  overlap  between  the  samples  used  in  the  within-domain  analyses 
(described  in  previous  chapters)  and  the  sample  used  in  the  present  cross-domain  analyses,  but 
also  a few  differences.  The  cross-domain  analyses  were  conducted  after  the  within-domain 
analyses,  and  additional  data  were  collected  while  the  within-domain  analyses  were  being 
conducted.  These  additional  data  were  included  in  the  cross-domain  analyses,  but  not  the  within- 
domain  analyses.  Thus,  the  raw  data  set  for  these  cross-domain  analyses  (before  screening  based 
on  missing  data  and  small  occupations)  contained  2487  respondents,  which  is  301  more 
respondents  than  were  available  for  the  within-domain  analyses.  However,  a stricter  screen  was 
also  applied  to  these  data:  occupations  were  only  included  in  the  cross-domain  analyses  if  at  least 
four  incumbents  (i.e.,  respondents)  were  available  for  each  of  the  nine  questionnaires  (i.e., 
domains).  Recall  that  occupations  were  included  in  the  within  domain  analyses  if  at  least  four 
incumbents  were  available  for  a particular  questionnaire.  The  additional  301  incumbents  were 
included  in  these  cross-domain  analyses  (in  part)  to  make  up  for  this  stricter  data  screen  and 
provide  as  many  occupations  as  possible  for  these  analyses. 

Only  respondents  who  had  completed  the  paper-and-pencil  versions  of  the  questionnaires 
were  included  in  the  present  analyses.  Questionnaires  were  excluded  if  more  than  ten  percent  of 
the  questions  were  not  answered.  Response  rates  for  each  questionnaire  were  examined 
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separately  and  individual  questionnaires,  not  entire  observations,  were  dropped.  Figure  13-1 
shows  the  29  occupations  included  in  the  cross-domain  analyses,  the  number  of  incumbents  from 
each  occupation  represented  in  this  data  set,  and  the  number  of  establishments  from  which  these 
incumbents  were  drawn.  Although  we  use  the  word  “incumbent”  to  describe  the  respondents  in 
this  sample,  7%  were  actually  the  supervisors  of  job  incumbents  and  20%  both  supervised  and 
performed  the  target  occupation. 

Most  respondents  completed  more  than  one  of  the  nine  questionnaires  and  a few 
completed,  as  many  as  five.  Thus,  a certain  percentage  of  the  data  available  for  comparing  any 
two  content  model  domains  are  based  on  responses  from  the  same  incumbents.  Table  13-1  shows 
the  percentage  of  data  for  any  given  comparison  (i.e.,  pair  of  domains)  that  is  based  on  responses 
from  the  same  incumbents. 

Approach 

The  initial  plan  for  the  cross-domain  analyses  focused  on  an  expanded  set  of  factor 
analyses,  incorporating  descriptors  from  several  domains  into  a single  analysis.  However,  based 
on  discussions  with  experts  from  the  various  content  domains,  our  approach  was  modified 
somewhat.  The  number  of  occupations  available  for  the  cross-domain  analyses  is  necessarily 
smaller  than  that  for  the  within-domain  analyses,  since  only  occupations  for  which  there  were 
data  for  all  domains  could  be  included.  The  number  of  descriptors  is  also,  by  definition,  larger 
when  more  than  one  domain  is  considered,  leading  to  serious  sample  size  problems  for  these 
analyses.  In  order  to  minimize  these  sample  size  problems  and  generate  more  defensible  results, 
we  developed  composites  to  summarize  information  concerning  the  descriptors  in  each  domain, 
and  relationships  between  these  composites  were  examined  across  domains.  To  generate  at  least 
some  preliminary  cross-domain  information  concerning  the  individual  descriptors,  we  also 
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conducted  a second  set  of  analyses  focused  on  examining  individual  descriptors  from  each  of 
several  domains  in  the  context  of  the  generalized  work  activities  (GW As).  Specifically,  the 
structure  of  the  relationships  between  descriptors  from  a particular  domain  and  the  GWAs  were 
examined.  Finally,  several  a priori  hypotheses  concerning  expected  relationships  between 
individual  descriptors  across  domains  were  generated  and  tested.  Each  of  these  analyses,  along 
with  the  results  obtained,  are  described  in  more  detail  in  the  sections  that  follow. 

Relationships  Between  Factor-Based  Composites 
The  entire  0*NET  content  model,  across  all  domains,  contains  a total  of  over  300 
descriptors,  and  the  sample  of  occupations  available  for  cross-domain  analyses  was  only  29  (with 
an  average  of  about  24  incumbents  per  occupation  within  each  domain).  In  order  to  minimize  the 
number  of  relationships  to  be  assessed,  we  identified  a smaller  set  of  composites  to  represent  the 
descriptors  in  each  domain.  These  composites  were  then  intercorrelated  across  occupations  and 
factor  analyzed  to  assess  the  overall  structure  of  the  0*NET  across  all  domains. 

Identification  of  Composites 

The  composites  were  identified  based  on  the  solutions  selected  in  the  within-domain 
principal  components  analyses,  described  in  the  relevant  chapter  for  each  domain,  and  then 
modified  based  on  rational  considerations.  First,  using  the  principal  components  solution  chosen 
for  each  domain  (presented  in  the  relevant  content  domain  chapters),  one  preliminary  factor- 
based  composite  was  developed  for  each  principal  component  extracted.  Items  were  assigned  to  a 
composite  if  they  loaded  at  least  .6  on  the  relevant  factor.  Items  with  substantial  loadings  on 
other  factors  (i.e.,  within  .2  of  the  primary  loading)  were  not  included.  Since  the  principal 
component  analyses  were  based  on  limited  samples  of  occupations  (between  30  and  37 
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occupations  for  each  content  domain),  these  factor-based  composites  were  then  refined  based  on 
rational  considerations. 

For  each  domain,  this  set  of  preliminary  composites  was  then  reviewed  by  the  individuals 
primarily  responsible  for  the  relevant  content  domain,  and  changes  were  made  based  on  the 
content  model  itself,  results  of  past  research,  and  other  rational  considerations.  In  general,  these 
modifications  were  minor,  and  involved  assigning  additional  descriptors  to  composites  (e.g., 
those  that  loaded  less  than  .60  on  the  relevant  composite).  For  the  abilities  domain  these 
revisions  were  somewhat  more  extensive.  The  0*NET  principal  components  analysis  for  this 
domain  yielded  a solution  that  was  somewhat  difficult  to  interpret,  and  past  research  in  the 
abilities  domain  has  typically  identified  a wide  variety  of  fairly  distinct  abilities.  Therefore,  a 
total  of  10  composites  was  identified  for  the  abilities  domain,  based  in  part  on' past  research  and 
in  part  on  the  principal  components  solution  for  the  0*NET  data.  Figure  13-2  shows  the 
composition  of  each  of  the  final  factor-based  composites.  For  two  domains— training,  education, 
licensure,  and  experience  and  organizational  context— no  composites  were  created,  because  very 
little  theory  is  available  to  interpret  the  correlations  between  higher-order  factors  in  these  two 
domains  and  those  from  the  other  domains.  Composite  scores  were  formed  by  calculating  mean 
scores  across  the  descriptors  included  in  each  of  the  composites.  Level  ratings  were  used  to  form 
scores  for  abilities,  skills,  knowledges,  work  styles,  and  generalized  work  activities. 

Occupational  values  scores  used  an  importance  scale,  and  work  context  ratings  were  varied. 

Composite  Intercorrelations 

Table  13-2  shows  the  intercorrelations  of  scores  on  these  factor-based  composites  across 
the  29  occupations  included  in  the  cross-domain  analyses.  Reliability  estimates  for  the 
composites  are  presented  on  the  diagonal  of  this  matrix.  These  were  calculated  by  computing  the 
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composite  scores  for  each  incumbent  providing  ratings,  and  then  assessing  the  interrater 
reliability  of  these  composite  ratings  across  occupations.  These  reliabilities  limit  the  extent  to 
which  the  composites  can  correlate,  and  are  provided  as  an  aid  in  interpreting  the  obtained 
correlational  results.  Note  that  for  some  of  the  composites  (e.g.,  Social  Comfort,  Structure,  and 
Detail  Orientation)  the  interrater  reliability  is  quite  low. 

Here  are  some  highlights  of  the  composite  correlations  across  domains.  Two  of  the  GWA 
composites-Working  with  Information  and  Working  with  and  Directing  Others-have  generally 
similar  patterns  of  correlations  with  the  skill,  ability,  and  work  style  composites.  For  example, 
they  both  correlate  highly  with  Cognitive  Abilities,  Cognitive  Skills,  and  Organizational  Skills, 
but  in  all  three  cases  the  correlation  is  slightly  higher  for  Working  with  Information.  These  two 
GWA  composites  also  correlate  with  Speech  Ability,  but  in  this  case  the  correlation  for  Working 
with  and  Directing  Others  is  slightly  higher.  Both  of  these  GW  As  also  correlate  at  about  the  same 
■ level  with  the  work  style  composite  labeled  Surgency/Achievement  Orientation.  These 
similarities  are  not  too  surprising  in  light  of  the  fact  that  these  two  GWA  composites  correlate 
.72  with  each  other.  Perhaps  this  reflects  the  fact  that  occupations  that  involve  working  with 
people  very  often  also  involve  working  with  information  (e.g.,  managerial  occupations). 

In  view  of  the  high  correlation  between  these  two  GW  As,  several  differences  in  the 
obtained  patterns  of  correlations  are  notable.  For  example,  Technical  Skills  and  Math  Ability 
correlate  significantly  with  Working  with  Information,  but  not  with  Working  with  and  Directing 
Others.  In  contrast,  Working  with  Others  correlates  significantly  with  the  work  style  composite 
People  Orientation  while  Working  with  Information  does  not.  The  differences  and  similarities  in 
the  patterns  of  correlations  across  occupations  for  these  two  GWA  composites  are  conceptually 
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sound  and  lend  a degree  of  construct  validity  to  these  GWA  descriptors  as  well  as  the  skill, 
ability  and  work  style  descriptors  with  which  they  correlate. 

The  pattern  of  correlations  for  the  third  GWA  composite^-Manual  and  Physical 
Activities— is  very  different,  and  lends  further  support  to  the  construct  validity  of  the  descriptors. 
For  example,  this  composite  correlates  positively  with  Physical  Ability,  Psychomotor  Ability, 
Dexterity,  and  Vision  and  Hearing  Ability,  while  the  other  two  GWAs  have  nonsignificant  and 
slightly  negative  correlations  with  these  four  ability  composites  across  occupations.  Manual  and 
Physical  Activities  correlates  significantly  with  only  one  of  the  three  skill  composites:  Technical 
Skills.  Again,  the  pattern  of  correlations  for  this  GWA  composite  with  the  skill  and  ability 
composites  are  intuitively  appealing  and  support  the  construct  validity  of  the  descriptors. 

There  is  a fair  degree  of  conceptual  overlap  between  the  descriptors  in  the  knowledges, 
skills,  and  abilities  domains,  so  obtaining  the  expected  correlations  between  these  descriptors 
across  occupations  was  an  important  source  of  construct  validity  evidence.  As  expected. 
Cognitive  Ability  correlated  highly  with  both  Cognitive  and  Organizational  Skills,  and  also 
correlated  significantly  with  Technical  Skills.  Spatial  Ability  correlated  positively  with  all  three 
skill  composites.  Interestingly,  Memory  ability  correlated  significantly  with  the  Cognitive  Skills 
composite  but  not  with  the  other  two  skill  composites.  Abilities  such  as  Physical  and 
Psychomotor  did  not  correlate  significantly  with  any  of  the  three  skill  composites. 

The  knowledge  composites  labeled  Science  and  Technology  and  Medicine  correlate 
significantly  with  Technical  Skill  requirements  and  to  a somewhat  lesser  extent  with 
Organizational  Skills.  This  is  not  surprising  because  occupations  in  science  and  technology  and 
in  medicine  often  require  technical  skills.  Both  the  Business  Administration  and  the  Art  and 
Humanities  knowledge  composites  correlated  with  Cognitive  and  Organizational  Skills,  but  not 
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with  Technical  Skills.  Clerical  and  Food  Production  knowledge,  on  the  other  hand,  were  not 
significantly  correlated  with  any  of  the  skill  composites,  perhaps  because  they  are  rather  specific, 
lower-level  knowledges.  Finally,  Law  Enforcement  knowledge  requirements  correlated  positively 
with  Cognitive  Skill  requirements,  perhaps  reflecting  the  cognitive  demands  involved  in  law 
enforcement  occupations. 

Relationships  between  ability  and  knowledge  requirements  are  generally  along  the  lines 
one  would  expect  as  well,  and  some  are  rather  interesting.  For  example,  occupations  that  require 
Clerical  knowledge  tend  not  to  require  Physical  Abilities  (r  = -.56).  This  brings  to  mind  someone 
who  spends  his  or  her  day  in  a chair  pushing  paper.  Similarly,  the  correlation  between  Business 
Administration  knowledge  and  Physical  Ability  requirements,  while  not  significant,  is  also 
negative.  Not  surprisingly.  Law  Enforcement  knowledge  requirements  are  correlated  with 
Psychomotor,  Vision  and  Hearing,  Spatial,  and  Attention  Ability  requirements,  probably  in  part 
due  to  the  fact  that  there  are  two  law  enforcement  occupations  in  the  sample.  Cognitive  Ability 
requirements  are  most  strongly  related  to  the  Arts  and  Humanities  knowledge  composite,  but  also 
related  to  several  other  knowledge  composites,  including  Law  Enforcement,  Medicine,  and 
Business  Administration.  Another  very  understandable  finding  is  that  Speech  Ability  is  most 
strongly  related  to  the  Business  Administration  and  Arts  and  Humanities  composites.  The 
abilities  most  strongly  related  to  medical  knowledge  requirements  are  Spatial  and  Attention.  The 
strong  correlation  between  Arts  and  Humanities  knowledges  and  Spatial  Abilities  is  somewhat 
surprising,  but  Spatial  Ability  seems  to  correlate  with  a broader  array  of  other  descriptors  than 
most  of  the  other  ability  composites.  Math  Ability  requirements  are  related  to  both  Science  and 
Technology  and  Business  Administration  knowledge  requirements.  Finally,  Food  Production 
knowledges  correlate  only  with  Physical  Abilities,  perhaps  a reflection  of  the  nature  of  the 
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occupations  in  our  sample.  Similarly,  Dexterity  is  only  correlated  with  the  knowledge  composite 
called  Medicine,  again  probably  a function  of  the  particular  occupations  in  our  sample. 

The  correlations  between  the  Surgency/ Achievement  Orientation  work  styles  composite 
and  both  Cognitive  and  Organizational  Skills  are  very  strong,  perhaps  suggesting  an  achievement 
related  constellation  of  job  requirements  that  covary  across  occupations.  Technical  Skill 
requirements  are  negatively  related  to  the  work  styles  composite  labeled  People  Orientation. 
Perhaps  occupations  that  require  technical  skills  also  tend  to  involve  working  alone,  or  perhaps 
interpersonal  interactions  are  not  a critical  aspect  of  these  occupations. 

Not  surprisingly,  there  is  a strong  relationship  between  the  GWA  composite  labeled 
Manual  and  Physical  Activities  and  a work  context  that  involves  Environmental  Factors. 
Similarly,  Working  with  Information  is  negatively  related  to  the  work  context  composite  Physical 
Activity,  while  Manual  and  Physical  Activities  are  positively  related  to  this  latter  aspect  of  work 
context.  Work  context  involving  Business/Office  is  negatively  related  to  Manual  and  Physical 
Activities,  and  positively  related  to  Working  with  and  Directing  Others.  Finally,  it  is  not  too 
surprising  that  Managerial  Relations  is  most  strongly  related  to  Working  with  and  Directing 
Others. 

As  might  be  expected,  occupational  values  related  to  Individual  Advancement  and  Career 
Accomplishment  are  correlated  with  the  Surgency/Achievement  Orientation  work  styles 
composite.  These  two  value  composites  are  also  fairly  strongly  related  to  the  GW  As  involving 
Working  with  Information  and  Working  with  and  Directing  Others  (but  not  Manual  and  Physical 
Activities)  and  also  to  Cognitive  and  Organizational  Skills  and  work  context  involving 
Managerial  Relations.  The  occupational  values  involving  Structure  are  positively  related  to 
working  in  a Business/Office  context.  Finally,  it  is  somewhat  surprising  that  the  occupational 
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values  related  to  Stability  are  quite  strongly  related  to  knowledge  requirements  in  the  Arts  and 
Humanities  and  Law  Enforcement  areas. 

It  should  be  kept  in  mind  that  these  cross-domain  comparisons  are  to  some  extent 
dependent  on  the  sample  of  occupations  included  in  the  present  analyses.  While  the  sample  does 
include  a range  of  occupations,  there  are  only  29  occupations  available,  and  it  is  unlikely  that 
such  a small  sample  is  totally  representative  of  occupations  in  general.  For  example,  this  sample 
includes  nurses  and  other  medical  professionals,  but  not  medical  doctors  and  some  of  the  other 
more  highly  educated  medical  professionals.  It  is  likely  that  if  a broader  sample  of  medical 
professionals  were  available,  some  of  the  correlations  involving  medical  knowledge  and  other 
descriptors  particularly  relevant  to  the  medical  profession  would  be  different.  In  addition,  a 
sample  of  29  does  not  provide  a great  deal  of  statistical  power.  It  is  likely  that  some  of  the 
correlations  that  were  not  significantly  different  from  zero  in  the  present  research  would  be 
significant  if  a larger  sample  were  available,  since  correlations  of  less  than  about  .37  are  not 
significant  at  the  p < .05  level  in  a sample  this  small. 

Although  most  of  the  data  used  in  each  comparison  (i.e.,  the  individual  correlations)  were 
collected  from  different  incumbents,  a certain  percentage  of  these  data  were  collected  from  the 
same  incumbents.  As  discussed  previously,  Table  13-1  shows  the  extent  to  which  this  occurred. 
For  several  pairs  of  domains,  virtually  all  of  the  data  came  from  the  same  sample  of  incumbents, 
and  this  is  primarily  because  the  number  of  questionnaires  and  the  length  of  the  questionnaires 
limited  how  many  different  combinations  of  domains  could  be  presented  to  individual 
incumbents.  There  is  some  reason  to  expect  that  data  collected  from  the  same  incumbents  will 
result  in  higher  correlations,  either  because  of  real  differences  in  jobs  within  an  occupation  or 
because  of  response  bias.  The  sample  available  in  the  present  research  was  not  large  enough  to 
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provide  a good  test  of  this  conjecture.  Thus,  correlations  between  GW  As  and  occupational  values 
should  be  interpreted  with  caution,  especially  as  they  compare  with  correlations  between  other 
pairs  of  domains. 

Factor  Analysis  of  Composites 

In  order  to  provide  a summary  of  the  structure  of  these  cross-domain  relationships,  the 
intercorrelations  of  the  factor-based  composites  were  factor  analyzed  using  principal  components 
analysis  with  a varimax  rotation.  A four  factor  solution  was  selected  based  on  the  eigenvalues 
and  the  meaningfulness  of  the  resulting  solutions.  The  factor  pattern  matrix  for  this  solution  is 
presented  in  Table  13-3. 

The  first  factor  has  strongest  loadings  for  composites  that  involve  working  with  or 
managing  others  and  achievement  or  accomplishment.  The  GWA  factor  that  involves  Working 
with  Information  loads  on  this  factor,  but  it  also  has  a fairly  high  loading  on  the  third  factor. 
Again,  the  high  correlation  between  this  GWA  and  Working  with  and  Directing  Others  appears 
to  have  affected  the  relationships  obtained.  Interestingly,  higher  levels  of  Cognitive  and 
Organizational  Skills  are  also  strongly  related  to  this  factor. 

The  second  factor  is  primarily  defined  by  Manual  and  Physical  Activities,  and 
Vision/Hearing,  Physical,  and  Dexterity  Abilities,  with  Law  Enforcement  knowledge  and 
Environmental  Factors  from  work  context.  The  third  factor  might  be  labeled  “General  Office” 
and  has  high  loadings  for  Clerical,  Speech,  and  Memory  Abilities,  as  well  as  Attention  and 
Cognitive  Abilities.  However,  these  latter  two  abilities  are  split  across  factors  two  and  one, 
respectively.  In  terms  of  work  context,  this  factor  has  a strong  positive  loading  for 
Business/Office  and  Structured/Machine  Operations,  and  a substantial  negative  loading  for 
Physical  Activity. 
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The  fourth  factor  is  somewhat  difficult  to  interpret,  because  it  is  defined  in  part  by 
positive  loadings  and  in  part  by  negative  loadings.  It  might  be  labeled  “Technical  versus 
Interpersonal.  Technical  Skills  and  Science  and  Technology  Knowledges  have  strong  positive 
loadings  on  this  factor.  On  the  other  hand,  the  work  style  labeled  Social  Orientation  (Work  Style 
Descriptor  #8)  and  work  context  involving  Dealing  with  the  Public  (Work  Context  Descriptor 
#60  have  substantial  negative  loadings.  The  GWA  composite  Working  with  and  Directing 
Others  also  has  a fairly  strong  negative  secondary  loading  on  this  fourth  factor. 

In  order  to  compare  the  structure  of  the  level  and  importance  ratings,  which  are  available 
for  each  descriptor  in  many  of  the  domains,  this  factor  analysis  was  conducted  again  using  the 
importance  ratings  where  appropriate.  For  five  of  the  seven  domains  included  in  this  factor 
analysis  both  level  and  importance  ratings  are  available.  For  the  remaining  two  domains-work 
context  and  occupational  values-only  a single  rating  was  collected  for  each  descriptor.  Thus,  this 
second  principal  component  analysis  included  the  same  data  as  the  initial  analyses  for  these  latter 
two  domains,  and  factor-based  composites  computed  using  the  importance  data  for  the  remaining 
five  domains.  The  results  were  virtually  identical  to  those  obtained  using  the  level  ratings,  which 
is  not  too  surprising  given  the  high  correlations  obtained  between  the  level  and  importance 
ratings  in  all  of  the  within  domain  analyses.  The  same  four  factors  emerged,  and  the  loadings 
were  very  similar  to  those  obtained  for  the  level  ratings. 

Analyses  of  Cross-Domain  Structure 

Analyses  using  the  factor-based  composites  are  necessarily  somewhat  limited.  For  most 
domains,  a subset  of  the  descriptors  could  not  be  included  on  any  composite.  In  addition,  the 
composites  provide  a fairly  broad  summary  of  each  content  domain  and  lack  the  detailed 
information  available  for  the  specific  descriptors.  Although  the  sample  sizes  were  limited  for  the 
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present  analyses,  we  made  an  initial  attempt  to  examine  the  cross-domain  relationships  in  more 
detail.  These  analyses  focused  on  relationships  between  the  generalized  work  activities 'and  each 
of  several  other  content  domains:  abilities,  work  styles,  occupational  values,  skills,  knowledges, 
and  work  context. 

The  first  set  of  analyses  described  here  were  aimed  at  understanding  the  relationships 
between  generalized  work  activities  (GW As)  and  abilities.  First,  we  computed  the  correlations 
between  each  of  the  individual  ability  descriptors  and  each  of  the  individual  GWA  descriptors, 
resulting  in  a correlation  matrix  where  the  columns  represented  abilities  and  the  rows  represented 
GWAs.  This  matrix  of  correlations  was  then  treated  as  the  data  of  interest,  and  we  intercorrelated 
the  columns.  That  is,  the  pattern  of  correlations  of  each  ability  descriptor  across  all  of  the  GWAs 
was  intercorrelated  with  the  pattern  of  GWA  correlations  for  each  of  the  other  ability  descriptors. 
This  resulted  in  a 52  by  52  intercorrelation  matrix.  The  sample  size  for  these  correlations  is  42, 
because  they  are  based  on  correlations  with  each  of  the  42  GWAs.  This  intercorrelation  matrix 
was  then  factor  analyzed,  in  order  to  obtain  a summary  of  the  relationships  between  the  ability 
descriptors,  as  they  relate  to  one  another  through  the  generalized  work  activities.  This  was  done 
using  principal  components  analysis  with  a varimax  rotation,  and  a three  factor  solution  was 
selected  based  on  the  eigenvalues  and  the  meaningfulness  of  the  resulting  solutions. 

Table  13-4  shows  the  factor  pattern  matrix  for  this  solution.  Essentially,  this  factor 
pattern  matrix  can  be  viewed  as  reflecting  the  structure  of  the  relationships  between  the  ability 
descriptors,  as  reflected  through  the  GWA  descriptors. 

Because  the  input  for  this  factor  analysis  was  the  set  of  correlations  between  abilities  and 
GWAs,  abilities  loading  highly  on  the  same  factor  are  expected  to  have  similar  patterns  of 
correlations  with  the  42  GWAs.  In  order  to  confirm  that  this  was  the  case,  we  plotted  the 
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correlations  between  four  abilities  from  the  same  factor  (Factor  1)  and  a representative  subset  of 
the  GWAs.  Nine  of  the  42  GWAs  were  selected  for  this  purpose.  One  GWA  was  selected  from 
each  of  the  nine  second-order  factors  in  the  GW  A content  model,  with  an  attempt  to  select  the 
most  reliable  GWAs  and  also  to  ensure  that  the  resulting  set  of  GWAs  adequately  represented  the 
three  principal  components  found  in  the  within-domain  analysis.  Figure  13-3  plots  the  nine 
correlations  for  each  of  the  four  abilities.  As  expected,  the  patterns  of  correlations  between  these 
four  abilities  and  the  nine  selected  GWAs  are  virtually  identical. 

We  then  chose  one  representative  ability  from  each  of  the  three  factors,  that  is,  an  ability 
with  a high  loading  on  the  relevant  factor  and  relatively  small  loadings  on  all  the  other  factors, 
and  plotted  the  correlations  between  each  of  these  abilities  and  the  same  nine  GWAs.  The  results 
are  shown  in  Figure  13-4.  These  patterns  of  correlations  are  useful  in  interpreting  the  factor 
solution  shown  in  Table  13-4,  because  the  loadings  of  the  abilities  on  each  of  the  three  factors  are 
based  on  their  patterns  of  correlations  with  the  GWAs. 

Table  13-4  shows  that  ability  descriptors  with  large  loadings  on  the  first  factor  include 
Far  Vision  (Ability  Descriptor  #42)  and  Spatial  Orientation  (Ability  Descriptor  #18).  Figure  13-4 
shows  that  the  first  factor  is  made  up  of  abilities  that  have  relatively  high  correlations  with 
Operating  Vehicles  (GWA  Descriptor  #20)  and  slightly  negative  correlations  with  Staffing 
Organizational  Units  (GWA  Descriptor  #41)  and  Analyzing  Data  (GWA  Descriptor  #9).  It  is 
sensible  to  expect  occupations  that  involve  operating  vehicles  to  require  far  vision  and  spatial 
orientation,  so  these  results  are  consistent  with  rational  expectations.  It  is  interesting  to  note  that 
many  of  the  physical  abilities  load  positively  on  this  first  factor  and  negatively  on  the  second 
factor.  The  second  factor  appears  to  be  made  up  of  high-level  cognitive  abilities,  such  as  Problem 
Sensitivity  (Ability  Descriptor  #7),  Inductive  Reasoning  (Ability  Descriptor  #9),  and  Originality 


1205 


Cross-Domain  Analysis  13-15 


(Ability  Descriptor  #6),  based  on  the  loadings  shown  on  Table  13-4.  Figure  13-4  shows  that 
variables  from  this  factor  correlate  positively  with  several  of  the  selected  GW  As  including 
Communicating,  External  (GWA  Descriptor  #28),  Identifying  Objects  (GWA  Descriptor  #2), 
Getting  Information  (GWA  Descriptor  #1),  and  Developing  Objectives  (GWA  Descriptor  #13). 
This  factor,  then,  appears  to  reflect  the  fact  that  many  cognitive  ability  requirements  tend  to  be 
correlated  with  certain  managerial  and  information  intensive  GWAs.  Table  13-4  shows  that  the 
third  factor  has  high  loadings  for  Perceptual  Speed  (Ability  Descriptor  #17)  and  Near  Vision 
(Ability  Descriptor  #41),  implying  activities  involving  close,  exact  work.  GWA  correlations  on 
Figure  13-4  show  that  these  abilities  are,  in  fact,  those  that  correlate  most  highly  with  Repairing, 
Mechanical  (GWA  Descriptor  #23). 

Analyses  were  conducted  for  five  other  domains,  parallel  to  the  analyses  just  described 
for  abilities.  For  each  of  these  domains,  occupation-level  correlations  between  the  individual 
descriptors  for  that  domain  and  the  individual  GWAs  were  computed,  the  columns  of  the 
resulting  correlation  matrices  were  intercorrelated  and  factor  analyzed,  and  correlations  between 
representative  descriptors  from  the  resulting  factors  and  each  of  the  nine  selected  GWAs  were 
plotted. 

Table  13-5  shows  the  three  factor  solution  for  work  styles,  Figures  13-5  and  13-6  plot  the 
correlations  with  the  nine  GWA  descriptors  for  work  styles  within  and  across  factors 
respectively.  These  results  are  not  as  clear  as  those  for  abilities,  but  there  are  some  interesting 
relationships.  The  first  factor  has  highest  loadings  for  some  of  the  more  cognitive  and 
achievement  oriented  work  styles  such  as  Analytical  Thinking  (Work  Style  Descriptor  #17), 
Persistence  (Work  Style  Descriptor  #2),  and  Achievement/Effort  (Work  Style  Descriptor  #1). 
GWAs  with  which  these  work  styles  correlate  most  highly  include  Developing  Objectives  (GWA 
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Descriptor  #13)  and  Identifying  Objects  (GWA  Descriptor  #2).  The  second  factor  is  defined 
primarily  by  work  styles  related  to  interpersonal  interactions,  such  as  Cooperation  (Work  .Style 
Descriptor  #6)  and  Concern  for  Others  (Work  Style  Descriptor  #7),  and  the  correlations  between 
these  work  styles  and  GWAs  appear  generally  low,  with  a moderate  negative  correlation  for 
Operating  Vehicles  (GWA  Descriptor  #20).  Perhaps  this  suggests  that  most  work  activities,  with 
the  exception  of  driving  vehicles,  require  at  least  some  interpersonal  interaction.  The  third  factor 
is  defined  by  a high  positive  loading  for  Attention  to  Detail  (Work  Style  Descriptor  #13)  and  a 
high  negative  loading  for  Energy  (Work  Style  Descriptor  #4).  This  could  suggest  that  work 
activities  requiring  a work  style  with  particularly  high  attention  to  detail  are  generally  sedentary. 
Again,  correlations  between  work  styles  from  this  factor  and  GWAs  are  generally  low,  and  the 
highest  correlation  shown  on  Figure  13-6  for  the  Attention  to  Detail  work  style  (Work  Style 
Descriptor  #13)  is  with  Identifying  Objects  (GWA  Descriptor  #2). 

The  four  factor  solution  for  occupational  values  is  shown  on  Table  13-6,  and  the  relevant 
GWA  correlations  are  shown  in  Figures  13-7  and  13-8.  The  first  factor  appears  to  be  defined  by 
achievement-related  values,  such  as  Responsibility  (Occupational  Value  Descriptor  #13), 
Achievement  (Occupational  Value  Descriptor  #2),  and  Ability  Utilization  (Occupational  Value 
Descriptor  #1).  Figure  13-8  shows  that  these  achievement  related  values  tend  to  be  negatively 
related  to  Operating  Vehicles  (GWA  Descriptor  #20)  and  Repairing,  Mechanical  (GWA 
Descriptor  #23)  and  positively  related  to  Communicating,  External  (GWA  Descriptor  #28), 
Developing  Objectives  (GWA  Descriptor  #13),  and  Getting  Information  (GWA  Descriptor  #1). 
The  second  factor  is  defined  by  Company  Policies  (Occupational  Values  Descriptor  #6)  and 
related  values,  and  these  values  have  very  small  positive  relationships  with  all  nine  representative 
GWAs,  with  the  largest  one  for  Developing  Objectives  (GWA  Descriptor  #13).  Perhaps  this 
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factor  reflects  systematic  differences  in  the  types  of  organizations  within  which  certain 
occupations  are  found,  but  it  is  difficult  to  draw  any  strong  conclusions  based  on  the  present 
analyses.  The  third  factor  has  highest  loadings  for  Security  (Occupational  Value  Descriptor  #14) 
and  Social  Service  (Occupational  Value  Descriptor  #15),  and  these  values  correlate  most  highly 
with  Operating  Vehicles  (GWA  Descriptor  #20).  The  fourth  factor  is  defined  by  Moral  Values 
(Occupational  Value  #1 1)  and  Coworkers  (Occupational  Value  Descriptor  #8),  and  the  values 
from  this  factor  have  small  negative  loadings  with  all  nine  GWAs  shown  on  Figure  13-8.  One 
GWA  not  included  in  this  figure— Handling  Objects  (GWA  Descriptor  #17)— correlated 
somewhat  more  strongly  with  Moral  Values  (Occupational  Values  Descriptor  #11)  (r  = .48). 
These  latter  two  factors  are  difficult  to  explain.  This  could  be  due  to  the  fact  that  the  occupational 
values  descriptors  are  generally  somewhat  less  reliable  than  those  in  most  of  the  other  domains, 
or  perhaps  because  there  is  less  theory  available  to  explain  relationships  between  work  activities 
and  workers’  occupational  values.  Also,  the  majority  of  the  data  for  these  two  domains  was 
collected  from  the  same  incumbents,  and  this  could  have  affected  the  obtained  results. 

Table  13-7  and  Figures  13-9  and  13-10  show  the  results  for  skills.  The  first  factor  is 
defined  by  higher-level  cognitive, 'managerial  and  interpersonal  skills.  These  skills  tend  to 
correlate  quite  highly  with  a variety  of  GWAs  including  Developing  Objectives  (GWA 
Descriptor  #13)  and  Directing  Subordinates  (GWA  Descriptor  #37).  The  second  factor  is  made 
up  of  more  technical  skills  including  Installation  (Skill  Descriptor  #28),  Troubleshooting  (Skill 
Descriptor  #35),  and  Repairing  (Skill  Descriptor  #36),  and  these  skills  have  very  high 
correlations  with  Repairing,  Mechanical  (GWA  Descriptor  #23)  and  somewhat  lower 
correlations  with  Operating  Vehicles  (GWA  Descriptor  #20).  The  final  factor  is  defined  almost 
exclusively  by  Programming  (Skill  Descriptor  #29)  skill  requirements.  Programming  has  its 
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highest  correlations  on  Figure  13-10  with  Getting  Information  (GWA  Descriptor  #1)  and 
Analyzing  Data  (GWA  Descriptor  #9).  Examination  of  GWAs  not  included  on  this  table  showed 
that  Programming  (Skill  Descriptor  #29)  has  substantial  correlations  with  the  GWAs  labeled 
Thinking  Creatively  (GWA  Descriptor  #11)  (r  = .63),  Interacting  with  Computers  (GWA 
Descriptor  #19)  (r  = .63)  and  Implementing  Ideas  (GWA  Descriptor  #22)  (r  = .57).  Apparently 
occupations  that  involve  programming  activities  have  a rather  distinct  pattern  of  skill 
requirements,  although  this  could  also  be  at  least  partly  an  artifact  of  the  limited  sample  of 
occupations  available  for  the  present  analyses. 

Table  13-8  and  Figures  13-1 1 and  13-12  show  the  results  for  knowledges.  The  first  factor 
is  defined  by  knowledges  related  to  business,  professional  and  managerial  activities.  Figure  13- 
12  shows  that  these  knowledges  correlate  with  several  GWAs  including  Getting  Information 
(GWA  Descriptor  #1),  Identifying  Objects  (GWA  Descriptor  #2),  and  Analyzing  Data  (GWA 
Descriptor  #9).  Apparently  these  are  knowledges  that  are  important  in  traditional  business 
settings.  The  second  factor  is  defined  by  a rather  odd  assortment  of  knowledges  including 
Foreign  Language  (Knowledge  Descriptor  #25)  and  Public  Safety  and  Security  (Knowledge 
Descriptor  #29).  These  knowledges  are  not  very  strongly  correlated  with  any  of  the  GWAs,  and  it 
is  difficult  to  identify  what  they  have  in  common.  The  third  factor  seems  to  involve  knowledges 
that  are  likely  to  be  related  to  manual  and  physical  activities,  such  as  Building  and  Construction 
(Knowledge  Descriptor  #12)  and  Mechanical  (Knowledge  Descriptor  #13).  This  is  confirmed  by 
the  fact  that  their  highest  correlations  are  with  Repairing,  Mechanical  (GWA  Descriptor  #23)  and 
they  also  correlate  with  Operating  Vehicles  (GWA  Descriptor  #20).  The  fourth  factor  is  basically 
science  and  medical  knowledge.  Correlations  between  these  knowledges  and  those  shown  on 
Figure  13-12  are  generally  small,  but  knowledge  of  Biology  (Knowledges  Descriptor  #17)  does 
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correlate  with  Monitoring  Processes  (GWA  Descriptor  #3)  (r  = .64),  Repairing,  Electronic  (GWA 
Descriptor  #24)  (r  = .64),  and  Teaching  Others  (GWA  Descriptor  #36)  (r  = .47).  The  sample  for 
these  analyses  includes  Medical  and  Clinical  Laboratory  Technologists  and  Registered  Nurses, 
and  perhaps  this  can  at  least  partially  explain  these  somewhat  surprising  results. 

Finally,  Table  13-9  and  Figures  13-13  and  13-14  show  the  results  for  work  context.  The 
first  factor  on  Table  13-9  appears  to  be  defined  by  hazardous  work  environments,  and  Figure  13- 
14  shows  that  these  work  context  descriptors  correlate  most  highly  with  Operating  Vehicles 
(GWA  Descriptor  #20)  and  Repairing,  Mechanical  (GWA  Descriptor  #23).  Not  too  surprisingly, 
these  aspects  of  work  context  are  also  correlated  with  Inspecting  Equipment  (GWA  Descriptor 
#4)  (r  = .56),  Performing  Physical  Work  Tasks  (GWA  Descriptor  #16)  (r  = .67)  and  Controlling 
Machines  (GWA  Descriptor  #18)  (r  = .58).  The  second  factor  is  defined  by  work  context 
descriptors  involving  managerial  relations,  and  these  descriptors  have  strong  correlations  with 
Directing  Subordinates  (GWA  Descriptor  #37)  and  Staffing  Organizational  Units  (Descriptor 
#41).  The  third  factor  focuses  on  interpersonal  job  requirements,  especially  those  that  involve 
dealing  with  the  public  or  difficult  interpersonal  situations.  Figure  13-14  shows  that  these  latter 
work  context  descriptors  tend  to  have  low  correlations  with  the  nine  representative  GW  As,  but  a 
negative  correlation  with  Repairing,  Mechanical  (GWA  Descriptor  #23).  GWA  correlations  not 
included  on  Figure  13-14  that  are  large  for  these  descriptors  include  Resolving  Conflicts  (GWA 
Descriptor  #32)  (r  = .52)  and  Working  with  the  Public  (GWA  Descriptor  #33)  (r  = .54).  Finally, 
the  fourth  factor  is  a combination  of  descriptors  such  as  Written  Reports  (Work  Context 
Descriptor  #2k),  Details  and  Completeness  (Work  Context  Descriptor  #32),  but  also 
Diseases/Infections  (Work  Context  Descriptor  #17),  perhaps  a reflection  of  the  fact  that  two  of 
the  occupations  included  in  our  sample  involve  nursing.  GWA  descriptors  not  shown  on  Figure 
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13-14  that  have  substantial  correlations  with  Written  Reports  (Work  Context  Descriptor  #2k) 
include  Documenting  Information  (GWA  Descriptor  #25)  (r  = .64),  Assisting  Others  (GWA 
Descriptor  #30)  (r  = .58)  and  Establishing  Relationships  (GWA  Descriptor  #29)  (r  = .50). 

A Priori  Cross-Domain  Relationships 

One  final  set  of  analyses  was  conducted  to  provide  an  additional  test  of  the  extent  to 
which  cross-domain  relationships  obtained  in  these  0*NET  prototype  data  are  consistent  with 
rational  and  theoretical  expectations.  This  involved  generating  and  testing  a series  of  hypotheses 
concerning  expected  correlations,  across  domains,  between  individual  descriptors.  We  began  by 
identifying  pairs  of  domains  for  which  strong  hypotheses  could  be  generated  concerning  expected 
relationships.  There  are  clear  similarities  across  a subset  of  the  skill  and  ability  descriptors.  For 
example,  one  of  the  ability  descriptors  is  Written  Comprehension  (Ability  Descriptor  #2)  and  one 
of  the  skill  descriptors  is  Reading  Comprehension  (Skill  Descriptor  #1).  It  would  be  truly 
surprising  if  occupations  rated  as  requiring  written  comprehension  were  not  also  rated  as 
requiring  reading  comprehension.  Similarly,  there  are  some  strong  conceptual  similarities 
between  certain  work  context  and  occupational  context  descriptors.  Another  pair  of  domains  for 
which  strong  hypotheses  could  be' generated  was  skills  and  GWAs;  similarly,  knowledges  have 
some  necessary  relationships  with  a subset  of  the  training  and  education  requirements.  Finally, 
evidence  from  past  research  is  available  concerning  relationships  between  GWAs  and  certain 
work  context  descriptors. 

Each  of  these  five  pairs  of  domains  was  examined  closely,  and  those  pairs  of  descriptors 
expected  to  correlate  across  domains  were  identified.  In  generating  these  cross-domain 
hypotheses,  we  focused  on  identifying  only  those  descriptors  for  which  very  strong  relationships 
were  expected.  In  other  words,  we  identified  those  pairs  of  descriptors  for  which  a finding  of  no 


Cross-Domain  Analysis  13-21 


relationship  would  be  very  unexpected  and  cast  doubt  on  the  construct  validity  of  the  descriptors 
involved.  In  order  to  provide  an  overall  assessment  of  the  extent  to  which  these  analyses  support 
the  construct  validity  of  the  descriptors  for  each  pair  of  domains,  we  compared  the  mean  of  these 
expected  correlations  to  the  mean  of  the  correlations  between  all  other  pairs  of  descriptors  across 
the  same  two  domains.  Correlations  were  first  converted  using  Fisher’s  r to  z transformation, 
then  averaged,  and  the  resulting  average  z-scores  were  converted  back  to  standard  correlation 
coefficients.  It  is  worth  noting  that  the  present  analyses  focused  on  subsets  of  the  most  obvious 
expected  correlations  between  descriptors,  and  many  other  pairs  of  domains  and  individual 
descriptor  correlations  remain  that  would  be  interesting  to  examine  in  the  context  of  past  research 
and  available  theory. 

Twelve  pairs  of  ability  and  skill  descriptors  were  identified  as  involving  very  similar 
content,  and  thus  hypothesized  to  correlate  highly.  The  average  of  these  twelve  correlations  was 
•.74,  while  the  overall  average  correlation  between  all  other  pairs  of  descriptors  across  these  two 
domains  was  only  .19.  A couple  of  examples  of  these  hypothesized  relationships  between  ability 
and  skill  descriptors  are  Written  Comprehension  (Ability  Descriptor  #2)  requirements  and 
Reading  Comprehension  (Skill  Descriptor  #1)  requirements  (r  = .84);  Originality  (r  = .81);  Idea 
Generation  (Skill  Descriptor  #21);  and  Problem  Sensitivity  (Ability  Descriptor  #7)  and  Problem 
Identification  (Skill  Descriptor  #17)  (r  = .61). 

For  the  work  context  and  organizational  context  domains,  there  were  nine  pairs  of 
descriptors  that  appeared  to  have  a great  deal  of  conceptual  similarity.  Although  these  two 
domains  are  relatively  distinct,  there  is  some  overlap.  For  example.  Decision  Latitude  (Work 
Context  Descriptor  #27)  is  an  aspect  of  an  individual  employee’s  work  context,  but 
Empowerment  (Organizational  Context  Descriptor  #1)  is  a construct  that  is  often  manipulated 
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and  studied  at  the  organizational  level.  Clearly  incumbents  who  report  a large  degree  of  Decision 
Latitude  would  also  be  expected  to  rate  their  jobs  high  on  Empowerment.  The  average 
correlation  across  the  nine  pairs  of  work  and  organizational  context  descriptors  identified  as 
overlapping  was  .59;  the  average  across  the  other  pairs  of  descriptors  was  only  .06.  The 
correlation  between  Empowerment  and  Decision  Latitude  was  .51.  The  latter  construct  correlated 
.67  with  Autonomy  (Organizational  Context  Descriptor  #2)  and  .47  with  a measure  of 
Decentralization  (Organizational  Context  Descriptor  #13).  An  organizational  context  variable, 
Percent  of  Time  Spent  in  Teams  (reflecting  a team  approach  to  organizational  structure; 
Organizational  Context  Descriptor  #26),  correlated  .57  with  a related  work  context  variable 
called  Work/Contribute  to  Teams  (Work  Context  Descriptor  #6e). 

The  conceptual  links  between  skills  and  generalized  work  activities  (GWAs)  are  not  quite 
as  strong  as  those  identified  between  skills  and  abilities,  but  24  pairs  of  skill  and  GWA 
descriptors  were  identified  that  were  expected  to  correlate  substantially.  The  average  of  these  24 
expected  correlations  was  .69,  while  the  average  of  all  of  the  remaining  skill-GWA  correlations 
was  .45.  For  example,  the  correlation  between  the  Speaking  (Skill  Descriptor  #4)  skill 
requirement  and  Communicating,  Internal  (GWA  Descriptor  #27)  was  .74.  Negotiation  Skill 
(Skill  Descriptor  #14)  requirements  correlated  .55  with  the  GWA  Selling  or  Influencing  (GWA 
Descriptor  #31).  The  skill  requirement  labeled  Operation  and  Control  (Skill  Descriptor  #32) 
correlated  .65  with  the  GWA  called  Controlling  Machines  (GWA  Descriptor  #18). 

It  is  reasonable  to  expect  that  if  an  occupation  requires  a higher  level  of  education  in  a 
particular  content  area  it  will  generally  require  a higher  level  of  knowledge  in  that  same  content 
area  as  well.  Thus,  we  compared  the  content  areas  for  15  education  requirements  with  the  content 
of  the  33  knowledge  descriptors  and  found  18  matches.  The  average  of  these  18  expected 
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correlations  was  .67,  and  the  average  of  the  remaining  correlations  was  .32.  For  example, 
knowledge  descriptors  labeled  English  Language  (Knowledge  Descriptor  #24)  correlated  .84 
with  level  of  education  required  in  English  Language/ Arts  (Education  Descriptor  #3c).  A few  of 
these  correlations  are  somewhat  lower  than  would  be  expected.  Language  (Education  Descriptor 
#3e)  education  requirements  correlated  only  .27  (not  significant)  with  Foreign  Language 
knowledge  (Knowledge  Descriptor  #25).  Still,  most  of  the  relationships  between  knowledge  and 
training  requirements  are  as  expected. 

Finally,  past  research,  especially  that  involving  the  Position  Analysis  Questionnaire, 
provides  information  concerning  relationships  between  certain  aspects  of  work  context  and  work 
tasks  (i.e.,  the  GW  As).  For  example,  work  activities  involving  controlling  machines  have  been 
found  to  be  related  to  certain  environmental  factors.  The  0*NET  content  model  contains  a large 
number  of  rather  specific  aspects  of  work  context,  so  we  used  the  work  context  factor-based 
composites  to  test  for  the  expected  relationships  with  individual  GW  A descriptors.  A total  of  15 
expected  relationships  were  identified,  and  the  average  of  these  correlations  was  .64;  the  average 
of  the  remaining  correlations  between  work  context  composites  and  GW  A descriptors  was  .11. 
The  correlation  between  Controlling  Machines  (GWA  Descriptor  #18)  and  Environmental 
Factors  was  .70.  The  correlation  between  the  Managerial  Relations  composite  and  the  GWA 
Directing  Subordinates  (GWA  Descriptor  #37)  was  .81.  In  general,  hypotheses  involving  the 
Business/Office  composite  were  not  as  strongly  supported  as  those  involving  the  other  work 
context  composites.  Possible  explanations  for  this  latter  finding  include  the  composition  of  this 
composite  (it  is  made  up  of  only  two  descriptors)  and  the  characteristics  of  the  occupations  in  the 
sample  (e.g.,  the  sampling  of  business  occupations  may  not  be  representative). 


Conclusions 
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Several  different  analytical  approaches  were  used  in  these  cross-domain  analyses,  and 
each  provides  a somewhat  different  perspective  on  the  relationships  between  descriptors  from  the 
various  0*NET  content  domains.  In  general,  these  results  strongly  support  the  construct  validity 
of  the  0*NET  descriptors  across  all  content  domains,  and  provide  some  interesting  insights 
concerning  these  cross-domain  relationships.  The  fact  that  most  of  the  cross-domain  comparisons 
involved  data  from  different  incumbents  strengthens  the  potential  generalizability  of  these 
findings.  All  of  the  tests  of  a priori  cross-domain  hypotheses  showed  that  where  strong 
correlations  were  expected,  strong  correlations  were  in  fact  obtained. 

In  general,  work  activities  involving  information  and  people  had  strong  correlations  with 
many  cognitive  ability  and  skill  requirements.  The  achievement-oriented  work  styles  and  those 
most  cognitively  oriented  were  also  strongly  related  to  activities  involving  information  and 
people,  as  well  as  to  cognitive  ability  and  skill  requirements.  Work  styles  involving  interpersonal 
•interactions  were  positively  correlated  with  activities  and  environments  involving  working  with 
others.  These  relationships  are  summarized  in  the  factor  analysis  results,  where  the  first  factor 
was  defined  by  descriptors  related  to  interpersonal  and  managerial  activities,  cognitive  skill 
requirements,  and  achievement-related  worker  characteristics.  This  factor  was  labeled 
"Management  and  Achievement."  Although  activities  involving  working  with  information  and 
working  with  people  had  generally  similar  patterns  of  correlations  with  descriptors  from  other 
domains,  the  differences  in  these  patterns  of  correlations  support  the  construct  validity  of  these 
composites.  For  example,  the  working  with  information  composite  was  more  strongly  related  to 
technical  skills  and  math  ability,  whereas  working  with  people  was  more  strongly  related  to  the 
people  oriented  work  style. 
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Manual  and  physical  work  activities,  on  the  other  hand,  were  correlated  with  technical 
skills  and  with  psychomotor  and  physical  ability  requirements.  Environmental  factors  from  the 
work  context  domain  also  tended  to  be  positively  correlated  with  manual  and  physical  activities 
and  related  worker  requirements.  In  fact,  manual  and  physical  activities,  physical  and 
psychomotor  abilities,  and  environmental  factors  defined  the  second  factor  in  the  factor  analysis. 

In  addition  to  obtaining  the  expected  relationships  across  domains,  the  present  analyses 
also  generally  showed  that  constructs  not  conceptually  related  do  not  correlate.  For  example, 
physical  and  psychomotor  ability  requirements  were  not  significantly  correlated  with  work 
activities  involving  information  or  people.  These  analyses  also  uncovered  a few  somewhat 
unexpected  but  conceptually  appealing  relationships.  For  example,  office  activities  and  related 
requirements  tended  to  be  negatively  correlated  with  physical  activities  and  related  worker 
requirements.  Technical  skill  requirements  were  negatively  related  with  a work  context  that 
involves  interacting  with  the  public,  and  finally,  law  enforcement  knowledge  requirements 
correlated  significantly  with  ability  requirements  such  as  psychomotor,  vision  and  hearing,  and 
spatial. 

The  more  detailed  analyses  of  relationships  between  individual  0*NET  descriptors  and 
GWAs  provide  some  additional  insights.  For  example,  one  of  the  ability  factors,  identified  based 
on  patterns  of  correlations  with  GWAs,  included  Far  Vision  (Ability  Descriptor  #42)  and  Spatial 
Orientation  (Ability  Descriptor  #18),  and  these  abilities,  in  turn,  tended  to  correlate  most  highly 
with  activities  such  as  Operating  Vehicles  (GWA  Descriptor  #20),  The  fact  that  Operating 
Vehicles  appears  to  be  related  to  a constellation  of  distinctly  different  abilities,  which  were 
actually  assigned  to  different  composites  in  the  composite-level  analyses,  could  only  be 
uncovered  in  these  descriptor-level  analyses.  Another  constellation  of  abilities  with  similar  GWA 
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correlations  included  Perceptual  Speed  (Ability  Descriptor  #17)  and  Near  Vision  (Ability 
Descriptor  #41),  and  these  abilities  correlated  most  highly  with  repair  activities  (e.g.,  Repairing, 
Mechanical;  GWA  Descriptor  #23).  Similar  analyses  for  work  styles  indicated  that  work  styles 
related  to  interpersonal  interactions  do,  in  fact,  tend  to  have  similar  patterns  of  GWA 
correlations.  For  skills,  these  analyses  revealed  that  Programming  Skill  (Skill  Descriptor  #29)  has 
a very  distinct  pattern  of  correlations  with  GW  As,  with  the  largest  correlations  involving  GW  As 
such  as  Thinking  Creatively  (GWA  Descriptor  #11)  and  Interacting  with  Computers  (GWA 
Descriptor  #19). 

In  summary,  results  of  the  cross-domain  analyses  provide  good  support  for  the  0*NET 
content  model  and  some  interesting  information  concerning  relationships  between  various 
occupational  requirements,  work  activities  and  other  characteristics.  Although  the  results  are 
limited  by  the  fact  that  the  sample  used  contains  only  29  occupations,  the  sample  did  include  a 
fairly  wide  variety  of  occupations  (see  Figure  13-1),  and  most  of  the  results  obtained  here  are 
likely  to  hold  up  when  larger,  more  representative  samples  are  available.  A larger  sample  will 
also  allow  for  more  detailed  analyses  of  the  structures  of  these  cross-domain  relationships,  as 
well  as  providing  the  power  to  detect  relationships  that  are  somewhat  weaker  but  still  important. 


Figure  13-1 

Twenty-Nine  Occupations  With  Four  or  More  Respondents  Across  all  Nine  Q*NET 
Questionnaires 


Occupation 

Code 

Occupation  Title 

Number  of 
Respondents 

Number  of 
Establishments 

15005 

Education  Administrators 

43 

12 

19005 

General  Managers  & Top  Executives 

147 

52 

25105 

Computer  Programmers 

22 

3 

31305 

Teachers,  Elementary  School 

84 

8 

32502 

Registered  Nurses 

97 

19 

32902 

Medical  & Clinical  Laboratory  Technologists 

19 

5 

49008 

Salespersons,  Except  Scientific  & Retail 

41 

11 

49011 

Salespersons,  Retail 

64 

12 

49021 

Stock  Clerks,  Sales  Floor 

33 

7 

49023 

Cashiers 

78 

24 

51002 

First  Line  Supervisors,  Clerical /Administrative 

174 

35 

53102 

Tellers 

20 

7 

53905 

Teachers'  Aides  & Assistants,  Clerical 

39 

14 

55108 

Secretaries,  Except  Legal  & Medical 

248 

6 7 

55305 

Receptionists  & Information  Clerks 

34 

14 

55338 

Bookkeeping,  Accounting,  & Auditing  Clerks 

94 

39 

55347 

General  Office  Clerks 

264 

67 

61005 

Police  & Detective  Supervisors 

39 

1 

63014 

Police  Patrol  Officers 

69 

4 

65008 

Waiters  & Waitresses 

60 

12 

65026 

Cooks,  Restaurant 

19 

7 

65038 

Food  Preparation  Workers 

89 

18 

66008 

Nursing  Aides,  Orderlies,  & Attendants 

56 

9 

67005 

Janitors  & Cleaners 

117 

34 

85132 

Maintenance  Repairers,  General  Utility 

110 

31 

87902 

Earth  Drillers,  Except  Oil  & Gas 

20 

1 

92974 

Packaging  & Filling  Machine  Operators 

40 

5 

97102 

Truck  Drivers,  Heavy  or  Tractor  Trailer 

41 

6 

97111 

Bus  Drivers,  Schools 

36 

4 

Figure  13-2 

Composition  of  Factor-Based  Composites  Used  in  Cross-Domain  Analyses 


Ability  Factor-Based  Composites 


1.  Physical 

Static  Strength;  Dynamic  Strength; 
Explosive  Strength;  Trunk  Strength; 
Stamina;  Gross  Body  Conditioning;  Gross 
Body  Equilibrium;  Dynamic  Flexibility; 
Extent  Flexibility 

2.  Psychomotor 

Rate  Control;  Control  Precision;  Multi-limb 
Coordination;  Reaction  Time;  Response 
Orientation;  Speed  of  Limb  Movement 

3.  Dexterity 

Manual  Dexterity;  Finger  Dexterity;  Arm- 
Hand  Steadiness;  Wrist-Finger  Speed 

4.  Vision/Hearing 

Depth  Perception;  Peripheral  Vision;  Night 
Vision;  Far  Vision;  Glare  Sensitivity; 

Visual  Color  Discrimination;  Sound 
Localization;  Hearing  Sensitivity;  Auditory 
Attention 


5.  Cognitive 

Deductive  Reasoning;  Originality; 

Fluency  of  Ideas;  Inductive  Reasoning; 
Problem  Sensitivity;  Information  Ordering; 
Category  Flexibility;  Written  Expression; 
Written  Comprehension;  Oral  Expression; 
Oral  Comprehension 

6.  Math 

Math  Reasoning;  Number  Facility 

7.  Spatial 

Visualization;  Spatial  Orientation; 
Flexibility  of  Closure;  Speed  of  Closure 

8.  Speech 

Speech  Clarity;  Speech  Recognition 

9.  Attention 

Perceptual  Speed;  Near  Vision;  Selective 
Attention;  Time  Sharing 

10.  Memory 
Memorization 


Generalized  Work  Activity 

1.  Manual  and  Physical  Activities 
Inspecting  Equipment;  Performing  Physical 
Work  Tasks;  Controlling  Machines; 

Operating  Vehicles;  Repairing,  Mechanical; 
Repairing,  Electronic;  Handling  Objects 

2.  Working  with  and  Directing  Others 
Establishing  Relationships;  Assisting 
Others;  Selling  or  Influencing;  Resolving 
Conflicts;  Working  with  the  Public; 
Coordinating  Others  Work;  Developing 
Teams;  Teaching  Others;  Directing 
Subordinates;  Developing  Others 


Factor-Based  Composites 

3.  Working  with  Information 

Getting  Information;  Identifying  Objects; 
Monitoring  Processes;  Estimating 
Characteristics;  Evaluating  Information; 
Processing  Information;  Analyzing  Data; 
Making  Decisions;  Thinking  Creatively; 
Using  Job  Knowledge;  Organizing  and 
Planning;  Interacting  with  Computers; 
Implementing  Ideas;  Interpreting 
Information;  Communicating,  Internal; 
Providing  Consultation 


Note.  Nine  Generalized  Work  Activities  are  not 
part  of  any  composite. 
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Composition  of  Factor-Based  Composites  Used  in  Cross-Domain  Analyses 


Work  Style  Factor-Based  Composites 

1. 

Surgency/ Achievement  Orientation 

3.  People  Orientation 

Achievement/Effort;  Persistence;  Initiative; 

Cooperation;  Concern  for  Others;  Social 

Energy;  Leadership  Orientation; 

Orientation;  Self-control;  Stress  Tolerance 

Independence;  Innovation;  Analytical 
Thinking 

2. 

Detail  Orientation 

Note.  Three  Work  Styles  are  not  part  of  any 

Attention  to  Detail 

composite. 

Knowledge  Factor-Based  Composites  | 

1. 

Arts  and  Humanities 

4.  Clerical 

Geography;  Therapy  and  Counseling; 
Foreign  Language;  Fine  Arts;  History  and 

Clerical 

Archeology 

5.  Medicine 

Chemistry;  Biology;  Medicine  and  Dentistry 

2. 

Science  and  Technology 

Production  and  Processing;  Engineering  and 

6.  Business  Administration 

Technology;  Design;  Building  and 

Sales  and  Marketing;  Customer  and 

Construction;  Mechanical;  Physics 

Personal  Service 

3. 

Law  Enforcement 

7.  Food  Production 

Public  Safety  and  Security;  Law, 

Food  Production 

Government,  and  Jurisprudence 

Note.  Thirteen  Knowledges  are  not  part  of  any 
composite. 

Occupational  Value  Factor-Based  Composites 

1.  Individual  Accomplishment 

Ability  Utilization;  Achievement;  Authority; 
Creativity;  Responsibility;  Variety; 
Autonomy 

2.  Structure 

Activity;  Independence  (negatively 
weighted);  Supervision  HR;  Supervision 
Tech 

3.  Stability 

Security;  Social  Status 


4.  Social  Comfort 

Co-workers;  Moral  Values;  Recognition; 
Working  Conditions 

5.  Career  Advancement 
Advancement;  Compensation;  Social 
Service  (negatively  weighted) 


Note.  One  Occupational  Value  is  not  part  of  any 
composite 


Figure  13-2  (Continued) 

Composition  of  Factor-Based  Composites  Used  in  Cross-Domain  Analyses 


Skill  Factor-Based  Composites 


1.  Cognitive  Skills 

Reading  Comprehension;  Active  Listening; 
Writing;  Speaking;  Critical  Thinking; 
Active  Learning;  Learning  Strategies; 
Monitoring;  Social  Perceptiveness; 
Coordination;  Persuasion;  Instructing; 
Problem  Identification;  Information 
Gathering;  Service  Orientation;  Information 
Organization;  Synthesis/Reorganization; 
Idea  Generation;  Idea  Evaluation; 
Implementation  Planning;  Solution 
Appraisal;  Time  Management;  Negotiation; 
Systems  Perceptions 


2.  Technical  Skills 

Mathematics;  Science;  Operations  Analysis; 
Technology  Design;  Equipment  Selection; 
Installation;  Programming;  Testing; 
Operation  Monitoring;  Operation  and 
Control;  Product  Inspection;  Equipment 
Maintenance;  Troubleshooting;  Repairing 

3.  Organizational  Skills 

Visioning;  Identification  of  Downstream 
Consequences;  Systems  Evaluation; 
Management  of  Financial  Resources; 
Management  of  Material  Resources; 
Management  of  Personnel  Resources; 
Identification  of  Key  Causes;  Judgment  and 
Decision  Making 


Work  Context  Factor-Based  Composites 


1.  Environmental  Factors 

Indoors,  Controlled  (negatively  weighted); 
Outdoors,  Exposed;  Outdoors,  Covered; 
Open  Vehicle/Equipment;  Enclosed 
Vehicle/Equipment;  Distracting  Noise; 
Extreme  Temperatures;  Poor  Lighting; 
Cramped  Work  Space;  Whole  Body 
Vibration;  High  Places;  Hazardous 
Conditions;  Hazardous  Equipment; 
Climbing  Ladders,  etc.;  Special  Safety 
Attire 

2.  Physical  Activity 

Sitting  (negatively  weighted);  Standing; 
Walking  or  Running;  Kneeling  or 
Crouching;  Keeping/Regaining  Balance; 
Bending/Twisting  Body 

3.  Health  and  Safety 

Written  Reports;  Physical  Aggression; 
Diseases;  Special  Uniform 


4.  Interacting  with  the  Public 
Public  Speaking;  Deal  with  Public 

5.  Managerial  Relations 
Face-to-Face  Groups;  Communications; 
Supervise/Develop  Others;  Persuade  or 
Influence;  Take  Opposing  Position; 
Coordinate/Lead  Activity;  Responsible 
for  Others  Work;  Accountable  for  Results; 
Decision  Latitude 

6.  Structured/Machine  Operations 

Level  of  Automation;  Accuracy/Exactness; 
Details  and  Completeness;  Repetitive 
Activities;  Structured  Tasks/Goals 
(negatively  weighted);  Machine  Driven 
Pace; 

7.  Business/ Office 

Business/Office  Clothes;  Deadline  Time 
Pressure  (negatively  weighted); 

Note.  Thirty-four  Work  Context  descriptors  are 
not  of  any  composite. 
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Correlations  with  Abilities 


Figure  13-3 

Correlations  Between  Representative  Abilities  and  Selected  Generalized  Work  Activities:  Within  a 
Single  Cross-Domain  Factor 
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Generalized  Work  Activities  (GWAs) 


Abilities 

—♦—Far  Vision  (42) 
—■—Spatial  Orientation  (18) 

- #-■  Peripheral  Vision  (45) 

A Response  Orientation  (27) 


GWAs 

1 - Getting  Information 

2 - Identifying  Objects 
9 - Analyzing  Data 

13  - Developing  Objectives 
20  - Operating  Vehicles 
23  - Repairing,  Mechanical 
28  - Communicating,  External 
37  - Directing  Subordinates 
41  - Staffing  Org.  Units 


A 


Correlations  with  Abilities 


Figure  13-4 

Correlations  Between  Representative  Abilities  and  Selected  Generalized  Work  Activities:  Across  All 
Cross-Domain  Factors 
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Generalized  Work  Activities  (GWAs) 


Abilities 

—♦—Far  Vision  (42) 

- ■ Time  Sharing  (21) 
—•—Perceptual  Speed  (17) 


GWAs 

1 - Getting  Information 

2 - Identifying  Objects 
9 - Analyzing  Data 

13  - Developing  Objectives 
20  - Operating  Vehicles 
23  - Repairing,  Mechanical 
28  - Communicating,  External 
37  - Directing  Subordinates 
41  - Staffing  Org.  Units 


Figure  13-5 

Correlations  Between  Representative  Work  Styles  and  Selected  Generalized  Work  Activities:  Within  a 
Single  Cross-Domain  Factor 
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Generalized  Work  Activities  (GWAs) 


Work  Style? 

♦ Innovation  (16) 

M Analytical  Thinking  (17) 

• Persistence  (2) 

A Initiative  (3) 


GW AS. 

1 - Getting  Information 

2 - Identifying  Objects 
9 - Analyzing  Data 

1 3 - Developing  Objectives 
20  - Operating  Vehicles 
23  - Repairing,  Mechanical 
28  - Communicating,  External 
37  - Directing  Subordinates 
41  - Staffing  Org.  Units 


ERiC 
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Figure  13-6 

Correlations  Between  Representative  Work  Styles  and  Selected  Generalized  Work-  Activities:  Across  All 
Cross-Domain  Factors 


WorK  Styles 

-♦—Innovation  (16) 

-■—Cooperation  (6) 

"• — Attention  to  Detail  (13) 


GWAs 

1 - Getting  Information 

2 - Identifying  Objects 
9 - Analyzing  Data 

13  - Developing  Objectives 
20  - Operating  Vehicles 
23  - Repairing,  Mechanical 
28  - Communicating,  External 
37-  Directing  Subordinates 
41  - Staffing  Org.  Units 


Figure  13-7 

Correlations  Between  Representative  Occupational  Values  and  Selected  Generalized  Work  Activities: 
Within  a Single  Cross-Domain  Factor 


Occupational  Values 

Working  Conditions  (20) 

-■—Ability  Utilization  (1) 
-•—Achievement  (2) 

-A— Autonomy  (21) 


QWAS. 

1 - Getting  Information 

2 - Identifying  Objects 
9 - Analyzing  Data 

13  - Developing  Objectives 
20  - Operating  Vehicles 
23  - Repairing,  Mechanical 
28  - Communicating,  External 
37  - Directing  Subordinates 
41  - Staffing  Org.  Units 
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Figure  13-8 

Correlations  Between  Representative  Occupational  Values  and  Selected  Generalized  Work  Activities: 
Across  All  Cross-Domain  Factors 
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Generalized  Work  Activities  (GW As) 


Occupational  Values 
-♦—Working  Conditions  (20) 
Company  Policies  (6) 
-♦—Security  (14) 

-A — Moral  Values  (11) 


GWAs 

1 - Getting  Information 

2 - Identifying  Objects 
9 - Analyzing  Data 

13  - Developing  Objectives 
20  - Operating  Vehicles 
23  - Repairing,  Mechanical 
28  - Communicating,  External 
37  - Directing  Subordinates 
41  - Staffing  Org.  Units 


Figure  13-9 

Correlations  Between  Representative  Skills  and  Selected  Generalized  Work  Activities:  Within  a Single 
Cross-Domain  Factor 


Skills 

—♦—Management  of  Personnel 
Resources  (46) 

B Time  Management  (43) 


-Management  of  Material 
Resources  (45) 

-Identification  of  Downstream 
Consequences  (39) 


ewAs. 

1 - Getting  Information 

2 - Identifying  Objects 
9 - Analyzing  Data 

1 3 - Developing  Objectives 
20  - Operating  Vehicles 
23  - Repairing,  Mechanical 
28  - Communicating,  External 
37  - Directing  Subordinates 
41  - Staffing  Org.  Units 
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Correlations  with  Skills 


Figure  13-10 

Correlations  Between  Representative  Skills  and  Selected  Generalized  Work  Activity-  Across  All  Cross- 
Domain  Factors 


Skills 


-Management  of  Personnel 
Resources  (46) 

-Installation  (28) 


-Programming  (29) 


GWAs 

1 - Getting  Information 

2 - Identifying  Objects 
9 - Analyzing  Data 

13  - Developing  Objectives 
20  - Operating  Vehicles 
23  - Repairing,  Mechanical 
28  - Communicating,  External 
37  - Directing  Subordinates 
41  - Staffing  Org.  Units 


Correlations  with  Knowledges 


Figure  13-11 

Correlations  Between  Representative  Knowledges  and  Selected  Generalized  Work  Activities:  Within  a 
Single  Cross-Domain  Factor 


Knowledges 

—♦—Mathematics  (14) 

—•—Computers  and  Electronics  (9) 
+ - Admin,  and  Management  (1) 

A English  Language  (24) 


GWAs 

1 - Getting  Information 

2 - Identifying  Objects 
9 - Analyzing  Data 

13  - Developing  Objectives 
20  - Operating  Vehicles 
23  - Repairing,  Mechanical 
28  - Communicating,  External 
37  - Directing  Subordinates 
41  - Staffing  Org.  Units 


Correlations  with  Knowledges 


Figure  13-12 

Correlations  Between  Representative  Knowledges  and  Selected  Generalized  Work  Activities:  Across  All 
Cross-Domain  Factors 


■ ■ * i i 

2 9 13  20  23  28  37  41 

Generalized  Work  Activities  (GW As) 


Knowledges 
♦ Mathematics  (14) 

-•—Foreign  Language  (25) 
-•—Building  and  Construction  (12) 
—A — Biology  (17) 


GWAs 

1 - Getting  Information 

2 - Identifying  Objects 
9 - Analyzing  Data 

13  - Developing  Objectives 
20  - Operating  Vehicles 
23  - Repairing,  Mechanical 
28  - Communicating,  External 
37  - Directing  Subordinates 
41  - Staffing  Org.  Units 


Figure  13-13 

Correlations  Between  Representative  Work  Context  and  Selected  Generalized  Work  Activities:  Within  a 
Single  Cross-Domain  Factor  , 


Work  Context 

♦ Health/Safety  of  Others  (7) 
—•—Contaminants  (1 5_04) 

• Extreme  Temperatures  (1 5_02) 
— A — Hazardous  Situations  (21_01) 


QWAs. 

1 - Getting  Information 

2 - Identifying  Objects 
9 - Analyzing  Data 

13  - Developing  Objectives 
20  - Operating  Vehicles 
23  - Repairing,  Mechanical 
28  - Communicating,  External 
37  - Directing  Subordinates 
41  - Staffing  Org.  Units 


Generalized  Work  Activities  (GW As) 


Figure  13-14 

Correlations  Between  Representative  Work  Context  and  Selected  Generalized  Work  Activities:  Across 
All  Cross-Domain  Factors 


Work  Context 

Health/Safety  of  Others  (7) 
-■—Responsible  Others'  Work  (8) 
-•—Social  Interaction  (4) 

—•r— Written  Reports  (2_11) 


GWAs 

1 - Getting  Information 

2 - Identifying  Objects 
9 - Analyzing  Data 

13  - Developing  Objectives 
20  - Operating  Vehicles 
23  - Repairing,  Mechanical 
28  - Communicating,  External 
37  - Directing  Subordinates 
41  - Staffing  Org.  Units 


Table  13-1 

Percentage  of  Data  Across  Pairs  of  Domains  Collected  from  Same  Incumbents 


Domain 

1 

2 

3 

4 

5 

6 

7 8 9 

1.  Abilities 

— 

2.  Generalized  Work  Activities 

10 

— 

3.  Organizational  Context 

9 

12 

— 

4.  Occupational  Values 

10 

93 

12 

— 

5.  Skills 

11 

11 

9 

11 

— 

6.  Training,  Licensure,  and  Experience 

10 

10 

12 

10 

10 

— 

7.  Work  Context 

10 

10 

12 

10 

12 

10 

— 

8.  Work  Styles 

11 

11 

12 

11 

10 

88 

11  - 

9.  Knowledge 

9 

12 

91 

12 

9 

12 

11  13  - 

Table  13-2 

Intercorrelations  of  Factor-Based  Composites  Across  Domains 


CD 

CD 

Cvi 


CO 


CM 


ON 


00 


VO 


in 


CO 


CM 


E 

o 

Q 

X) 

C/3 

<d 

•w 
• ^H 

C/3 

o 

CX 

E 

o 

U 


oo 


O CM 
ON  -H 


O <N  O 
on  r-  cm 


2 a 

V o 
x \3 


<D 


u 

< 

U. 

o 

£ 

TD 

<D 

N 

E 

<d 

a 

v 

o 


c 

.2 

E 

E 

U 

c2 

£ 


O > 

OX)  u 

.s  < 


o 

<D 


> ■ 5 -5 


u £ 
_ « £ 

£ £ ■§ 

OX)  OX)  ^ 

.5  .S  13 
J3  3 

Ur  Ur  C 

O o e3 

£ £S 

^ CM  CO 


ON 

00 


vo  r- 

On  i 


O m ^ 
On  '-i  Tf 


O vo  — ■ CM 
OO  CN  O 1 


CO  CO  o 00  CO 
CN  O — < -H  (N 


O ^ h m a\ 
On  ^-h  co  in  — i o 

w i*  r i*  * i* 

CO  o VO  Tt 

On  Tt  CM  O CO  CM  O 

w ’ * * r i’ 

O Tf  CM  CO  VO  in  (N 

oo  in  - - m n (N 

i*  * r 

^ (N  in  On  id  co  h« 

O (N  h o co  in 

»*  " i*  * * * 

O co  vo  in  on  ^ 

cm  in  vo  o co  o cm 


C/3 

c 

o 


On 


^ 00 
On  in 


3-  - rt 
On  m 00 


VO  Tt  vo 

CM  CO  O 


— ' CO  CM 
CM  -H  o 


vo  o VO 

CO  CO  CM 


cm  r-  r-* 
o cm  o 


CM  00  On 
co  r^ 


r-**  Tt 
CO  o CM 


Tt  in  in 
O co  — < 


r-  cm  in 
o in  o 


m oo  oo 
h o vo 


m 

oo 


**t  CM 


a 

ill 

00  00  o 

*>  3 C3 

.Con 

-*— » * .PN 

•g  G C 

§>  o S> 

O O i 

U H O 

£3  ^ CM  CO 


O on  h ON  CO  rt  Tf 
co  co  o co  in  — ' 


cm  vo  — i vo  co  ^ vo 
^ in  ^ o in  o O 


r-  ^ cm  On  in  co 
^ (M  Tf  cm  co  in  co 


in  vo  on  ^ in  o O 

CM  CM  CM  ^ o 'O  ^ 


cMcMONTfr-cor- 
pH  pH  in  u Tt  -H  ^ 


O O cm  cm  co  Tt  o 

CO  CM  CM  VO  o Tf  CM 


^ ^ On  On  in 

O co  — h co  O O 


oo  oo  Tt  in  in  in  cm 

CO  CM  ^ CO  Tf  O 


co  vo  in  in  on  on 
O CM  O vo  O M" 


incovooNvoinr- 
CM  in  Tf  CO  » CM 


co  rt  cm  r*-  co  co  ^ 
O in  cm  tj-  co  co 


CM  co  cm  oo  r-  in  On 

m o vo  o cm  in  o 


vo  00  O CM  O CO  00 
CM  O CO  CO  CM  Tf  CM 


c/3 

V 

OX) 


c 

o 

E 

3 

X 

T3 

C 

C5 


>% 

OX) 

JO 

o 

c _ 

43  G 


WAJ  +-* 

■8  5 


u 

<D  e 

H E 
L«  « 

T3  o 
C fe 
<0 

CL)  C 

g w 

.2  £ 

O cd 

CO  J 


c 

o 


c s 
*3  O 

E 


ID 

—J  C 


T3 

< 

C/3 

c/3 


O 

3 

T3 

O 


3 o § fi- 

ll ! 1 

US'Bfc 


- - - 2 ^ Zi 


^ m vo 


h oo  On  O 


CO 


S3 

ERIC 


\n 

co 

C\2 


Table  13-2  (continued) 

Intercorrelations  of  Factor-Based  Composites  Across  Domains 
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Table  13-2  (continued) 

Intercorrelations  of  Factor-Based  Composites  Across  Domains 
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Table  13-3 

Principal  Components  Analysis  Pattern  Matrix:  Cross-Domain  Factor-Based  Composites 


Domain  and  Composite  Label 

WC:  Managerial  Relations 
SK:  Cognitive  Skills 
SK:  Organizational  Skills 
WS:  Surgency,  Achievement  Orientation 
GWA:  Working  with/Directing  Others 
OV:  Individual  Accomplishment 
OV:  Career  Advancement 
GWA:  Working  with  Information 
KN:  Business  Administration 
OV:  Stability 
KN:  Arts  and  Humanities 
OV:  Social  Comfort 
AB:  Psychomotor 
AB:  Vision/Hearing 
AB:  Physical 
AB:  Dexterity 
WC:  Environmental  Factors 
GWA:  Manual  and  Physical  Activities 
KN:  Law  Enforcement 
AB:  Spatial 
WC:  Health  and  Safety 
KN:  Medicine 
KN:  Food  Production 
AB:  Attention 
AB:  Memory 
AB:  Cognitive 
AB:  Speech 
KN:  Clerical 
WC:  Business/Office 
AB:  Math 

WC:  Structured/Machine  Operations 
OV:  Structure 
WC:  Physical  Activity 
SK:  Technical  Skills 
KN:  Science  and  Technology 
WS:  Detail  Orientation 
WS:  People  Orientation 
WC:  Interacting  with  Public 
Percent  of  V ariance 
Eigenvalues 


Factor 


F2 

F3 

F4 

Communality 

.11 

-.03 

.04 

.79 

.05 

.21 

.23 

.92 

-.01 

.39 

.05 

.90 

-.04 

.05 

.07 

.70 

.08 

.25 

-.41 

.86 

-.15 

;06 

-.05 

.65 

.06 

-.13 

-.11 

.62 

-.23 

.41 

.09 

.79 

-.24 

.28 

-.18 

.50 

.40 

.13 

-.28 

.58 

.44 

.32 

-.18 

.56 

-.31 

.34 

-.04 

.33 

.92 

-.05 

-.03 

.87 

.91 

.11 

-.16 

.88 

.88 

-.35 

-.09 

.91 

.82 

.19 

.16 

.84 

.80 

-.29 

.15 

.78 

.73 

-.33 

.42 

.82 

.65 

.26 

-.35 

.77 

.61 

.55 

.09 

.88 

.54 

.04 

-.31 

.40 

.48 

.34 

.33 

.51 

.40 

-.17 

.12 

.21 

.49 

.80 

.13 

.91 

.12 

.79 

-.09 

.72 

.12 

.75 

.09 

.88 

.08 

.74 

-.38 

.77 

-.33 

.70 

.00 

.60 

-.26 

.69 

-.38 

.72 

-.09 

.66 

.40 

.68 

.18 

.64 

.38 

.64 

-.20 

.43 

.22 

.29 

.42 

-.62 

.06 

.57 

.30 

.14 

.75 

.83 

.54 

.10 

.56 

.68 

-.15 

.14 

.26 

.11 

.01 

-.03 

-.60 

.40 

.02 

.20 

-.67 

.58 

20 

17 

9 

7.57 

4.10 

3.24 

FI 

.88 

.87 

.87 

.83 

.80 

.79 

.77 

.75 

.58 

.57 

.47 

.35 

-.14 

-.08 

-.10 

-.34 

.16 

.01 

.40 

.44 

.10 

.24 

-.07 

.08 

.26 

.55 

.27 

-.4 

.18 

.29 

-.22 

.08 

-.09 

.39 

.28 

.07 

.20 

.28 

21 

10.53 


Table  13-3  (continued) 

Principal  Components  Analysis  Pattern  Matrix:  Cross-Domain  Factor-Based  Composites 

Note,  N = 29.The  correlation  matrix  was  based  on  means  calculated  at  the  occupation  level.  FI  = 
Management  and  Achievement,  F2  = Manual  and  Physical,  F3  = General  Office,  F4  = Technical 
versus  Interpersonal.  These  loadings  are  based  on  an  orthogonal  varimax  rotation.  Domains  are 
abbreviated:  WC  = Work  Context;  SK  = Skills;  WS  = Work  Styles;  GWA  = Generalized  Work 
Activities;  OV  = Occupational  Values;  KN  = Knowledges;  AB  = Abilities. 


Table  13-4 

Principal  Components  Analysis  Pattern  Matrix  for  Cross-Domain  Analysis  of  Correlations  with 
Generalized  Work  Activities:  Abilities 


Factor 


Ability  Descriptors 

FI 

F2 

F3 

Communality 

1.  Oral  Comprehension 

-.48 

.84 

-.18 

.98 

2.  Written  Comprehension 

-.46 

.87 

-.02 

.97 

3.  Oral  Expression 

-.39 

.90 

-.17 

.99 

4.  Written  Expression 

-.46 

.86 

-.20 

.99 

5.  Fluency  of  Ideas 

-.44 

.85 

-.27 

.99 

6.  Originality 

-.38 

.87 

-.13 

.93 

7.  Problem  Sensitivity 

-.15 

.96 

-.12 

.95 

8.  Deductive  Reasoning 

-.41 

.90 

-.01 

.97 

9.  Inductive  Reasoning 

-.24 

.93 

-.06 

.93 

10.  Information  Ordering 

-.45 

.84 

.26 

.98 

11.  Category  Flexibility 

-.51 

.77 

-.16 

.88 

12.  Math  Reasoning 

-.66 

.67 

,06 

.88 

13.  Number  Facility 

-.67 

.71 

.07 

.96 

14.  Memorization 

-.37 

.89 

-.14 

.95 

15.  Speed  of  Closure 

-.43 

.88 

-.02 

.97 

16.  Flexibility  of  Closure 

-.20 

.88 

.21 

.86 

17.  Perceptual  Speed 

.28 

.17 

.92 

.96 

18.  Spatial  Orientation 

.95 

-.26 

.09 

.97 

19.  Visualization 

.37 

.30 

.56 

.54 

20.  Selective  Attention 

.11 

.88 

.35 

.90 

21.  Time  Sharing 

-.17 

.96 

.02 

.96 

22.  Arm-hand  Steadiness 

.73 

-.40 

.51 

.95 

23.  Manual  Dexterity 

.71 

-.51 

.46 

.98 

24.  Finger  Dexterity 

.56 

-.54 

.61 

.98 

25.  Control  Precision 

.70 

-.46 

.48 

.93 

26.  Multi-limb  coordination 

.78 

-.53 

.28 

.96 

27.  Response  Orientation 

.92 

-.24 

.26 

.98 

28.  Rate  Control 

.82 

-.44 

.24 

.92 

29.  Reaction  Time 

.87 

-.35 

.31 

.97 

30.  Wrist  Finger  Speed 

.19 

-.38  • 

.69 

.66 

31.  Speed  of  Limb  Movement 

.83 

-.48 

.21 

.96 

32.  Static  Strength 

.78 

-.56 

.23 

.98 

33.  Explosive  Strength 

.88 

-.43 

.16 

.98 

34.  Dynamic  Strength 

.82 

-.50 

.20 

.96 

35.  Trunk  Strength 

.75 

-.59 

.28 

.99 

36.  Stamina 

.81 

-.53 

.16 

.96 

37.  Extent  Flexibility 

.71 

-.64 

.29 

.99 

38.  Dynamic  Flexibility 

.74 

-.63 

.18 

.99 

39.  Gross  Body  Coordination 

.80 

-.54 

.20 

.97 

■ v *•  ’• 


Table  13-4  (continued) 


Generalized  Work  Activities:  Abilities 
Ability  Descriptors 

FI 

F2 

Factor 

F3 

Communality 

40.  Gross  Body  Equilibrium 

.74 

-.63 

.14 

.97 

41.  Near  Vision 

.48 

.14 

.73 

.78 

42.  Far  Vision 

.96 

.14 

-.02 

.94 

43.  Visual  Color  Discrimination 

.67 

-.35 

.57 

.90 

44.  Night  Vision 

.92 

-.32 

.16 

.96 

45.  Peripheral  Vision 

.93 

. -.28 

.20 

.99 

46.  Depth  Perception 

.90 

-.35 

.22 

.99 

47.  Glare  Sensitivity 

.88 

-.16 

.31 

.90 

48.  Hearing  Sensitivity 

.88 

-.27 

.36 

.97 

49.  Auditory  Attention 

.91 

-.24 

.27 

.96 

50.  Sound  Localization 

.91 

-.28 

.28 

.98 

51.  Speech  Recognition 

-.08 

.82 

-.28 

.75 

52.  Speech  Clarity 

-.28 

.88 

-.30 

.94 

Percent  of  Variance 

44 

39 

11 

Eigenvalues 

38.41 

7.75 

2.47 

Note.  N = 42.  The  correlation  matrix  was  based  on  occupation-level  correlations  between 
abilities  and  generalized  work  activities.  FI  = Operating  Vehicles  and  Physical  Tasks  with  .... 
Physical  and  Perceptual  Abilities,  F2  = Cognitively  Demanding  Tasks  with  High  Level  Cognitive 
Abilities,  F3  = Manual/Repair  Tasks  with  Perceptual/Psychomotor  Abilities.  These  loadings  are 
based  on  an  orthogonal  varimax  rotation. 
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Table  13-5 

Principal  Components  Analysis  Pattern  Matrix  for  Cross-Domain  Analysis  of  Correlations  with 
Generalized  Work  Activities:  Work  Styles 


Factor 


1 

Work  Style  Descriptors 

FI 

F2 

F3 

Communality 

1.  Achievement/Effort 

.91 

.33 

-.06 

.94  • 

2.  Persistence 

.95 

.28 

.07 

.98 

3.  Initiative 

.95 

.24 

-.10 

.96 

4.  Energy 

.31 

.32 

-.73 

.73 

5.  Leadership  Orientation 

.86 

.45 

-.13  ' 

.97 

6.  Cooperation 

-.03 

.91 

-.04 

.83 

7.  Concern  for  Others 

.27 

.91 

-.13 

.92 

8.  Social  Orientation 

.20 

.91 

-.28 

.94 

9.  Self-control 

-.19 

.92 

-.24 

.94 

10.  Stress  Tolerance 

.59 

.76 

.15 

.96 

11.  Adaptability/Flexibility 

.81 

.54 

.08 

.96 

12.  Dependability 

.50 

.80 

-.07 

.89 

13.  Attention  to  Detail 

.29 

-.05 

.82 

.77 

14.  Integrity 

.50 

.81 

.02 

.90 

15.  Independence 

.80 

-.08 

.06 

.65 

16.  Innovation 

.98 

.09 

-.08 

17.  Analytical  Thinking 

.95 

-.01 

.22 

.95 

Percent  of  Variance 

45 

36 

9 

Eigenvalues 

10.23 

3.81 

1.23 

Note.  N = 42.  The  correlation  matrix  was  based  on  occupation-level  correlations  between  work  styles  and 
generalized  work  activities.  FI  = Demanding/Cognitive  Tasks  with  Cognitively  Oriented  and 
Achievement-Related  Work  Styles,  F2  = Working  with  Others  with  Interpersonal  Work  Styles,  F3  = 
Detail  Oriented  Work  Tasks  with  High  Attention  to  Detail  and  Low  Energy.  These  loadings  are  based  on 
an  orthogonal  varimax  rotation. 


Table  13-6 

Principal  Components  Analysis  Pattern  Matrix  for  Cross-Domain  Analysis  of  Correlations  with 
Generalized  Work  Activities:  Occupational  Values 


Factor 


Occupational  Value  Descriptors 

FI 

F2 

F3 

F4 

Communality 

1 . Ability  Utilization 

.92 

.36 

-.02 

-.07 

.99 

2.  Achievement 

.89 

.39 

.05 

-.04 

.96 

3.  Activity 

.79 

.37 

-.27 

.11 

.85 

4.  Advancement 

.74 

.55 

-.04 

-.09 

.85 

5.  Authority 

.78 

.41 

.40 

-.08 

.95 

6.  Company  Policies 

.30 

.85 

.10 

.06 

.83 

7.  Compensation 

.39 

.67 

-.17 

-.31 

.72 

8.  Co-workers 

.12 

.08 

.17 

.85 

.78 

9.  Creativity 

.87 

.44 

.15 

-.10 

.98 

10.  Independence 

-.18 

-.81 

.01 

.10 

.69 

11.  Moral  Values 

-.12 

-.21 

-.17 

.90 

.91 

12.  Recognition 

.62 

.73 

-.05 

-.06 

.92 

13.  Responsibility 

.78 

.30 

.27 

-.31 

.86 

14.  Security 

.50 

-.08 

.89 

-.30 

.89 

15.  Social  Service 

.23 

-.16 

.82 

.40 

.91 

16.  Social  Status 

.84 

.09 

.47 

-.09 

.95 

17.  Supervision  Human  Resources 

.42 

.76 

-.22 

.10 

.81 

18.  Supervision  Technical 

-.58 

.01 

.66 

.42 

.95 

19.  Variety 

.85 

.30 

.40 

-.01 

.98 

20.  Working  Conditions 

.91 

.15 

-.13 

.18 

.90 

21.  Autonomy 

.88 

.30 

-.22 

.14 

.93 

Percent  of  Variance 

13 

21 

13 

11 

Eigenvalues 

45.72 

10.67 

6.60 

5.56 

Note^  N = 42.  The  correlation  matrix  was  based  on  occupation-level  correlations  between  abilities  and 
generalized  work  activities.  FI  = Achievement-Related  Values  with  Information/Interpersonal  Tasks,  F2 
= Company  Policy-Related  Values,  F3  = Security/Social  Service,  F4  = Moral  Values/Coworkers.  These 
loadings  are  based  on  an  orthogonal  varimax  rotation. 
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Table  13-7 

Principal  Components  Analysis  Pattern  Matrix  for  Cross-Domain  Analysis  of  Correlations  with 
Generalized  Work  Activities:  Skills 


Factor 


Skill  Descriptors 

FI 

F2 

F3 

Communality 

1.  Reading  Comprehension 

.86 

-.26 

.39 

.95 

2.  Active  Listening 

.82 

-.47 

.29 

.97 

3.  Writing 

.86 

-.32 

.36 

.97 

4.  Speaking 

.90 

-.30 

.27 

.96 

5.  Mathematics 

.81 

.08 

.52 

.94 

6.  Science 

.67 

.47 

.47 

.89 

7.  Critical  Thinking 

.91 

-.19 

.37 

.99 

8.  Active  Learning 

.90 

.08 

.39 

.98 

9.  Learning  Strategies 

.93 

-.06 

.19 

.91 

10.  Monitoring 

.95 

-.18 

.20 

.97 

11.  Social  Perceptiveness 

.88 

-.41 

.00 

.95 

12.  Coordination 

.94 

-.09 

.29 

.98 

13.  Persuasion 

.92 

-.33 

.15 

.97 

14.  Negotiation 

.92 

-.36 

.05 

.98 

15.  Instructing 

.86 

.06 

.27 

.81 

16.  Service  Orientation 

.86 

-.46 

-.05 

.95 

17.  Problem  Identification 

.89 

-.13 

.40 

.97 

18.  Information  Gathering 

.90 

-.18 

.34 

.97 

19.  Information  Organization 

.88 

-.11 

.44 

.99 

20.  Synthesis/Reorganization 

.89 

-.15 

.43 

.99 

21.  Idea  Generation 

.92 

-.07 

.36 

.98 

22.  Idea  Evaluation 

.91 

-.14 

.37 

.99 

23.  Implementation  Planning 

.89 

-.14 

.42 

.99 

24.  Solution  Appraisal 

.92 

-.20 

.32 

.99 

25.  Operations  Analysis 

.80 

.13 

.55 

.97 

26.  Technology  Design 

.68 

.50 

.52 

.98 

27.  Equipment  Selection 

.66 

.70 

.19 

.95 

28.  Installation 

-.20 

.95 

.01 

.94 

29.  Programming 

.43 

.28 

.81 

.91 

30.  Testing 

.16 

.87 

.46 

.99 

31.  Operation  Monitoring 

-.35 

.89 

-.23 

- .97 

32.  Operation  and  Control 

-.32 

.91 

-.16 

.95 

33.  Product  Inspection 

.15 

.97 

.07 

.96 

34.  Equipment  Maintenance 

-.41 

.85 

-.28 

.97 

35.  Troubleshooting 

-.05 

.95 

.30 

.99 

36.  Repairing 

-.30 

.94 

.02 

.98 

37.  Visioning 

.83 

.26 

.10 

.76 

38.  Systems  Perception 

.93 

.31 

.13 

.97 

39.  Identification  of  Downstream 

.96 

-.10 

.10 

.94 

Consequences 

40.  Identification  of  Key  Causes 

.96 

-.12 

.21 

.97 

41.  Judgment  and  Decision  Making 

.93 

-.23 

.24 

.98 
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Table  13-7  (continued) 

Principal  Components  Analysis  Pattern  Matrix  for  Cross-Domain  Analysis  of  Correlations  with 
Generalized  Work  Activities:  Skills  — 


Factor 


Skill  Descriptors 

FI 

F2 

F3 

Communality 

42.  Systems  Evaluation 

.94 

.10 

.25 

.96 

43.  Time  Management 

.97 

-.16 

.15 

.99 

44.  Management  of  Financial  Resources 

.94 

-.19 

.06 

.92 

45.  Management  of  Material  Resources 

.97 

.16 

-.09 

.97 

46.  Management  of  Personnel  Resources 

.97 

-.07 

-.11 

.96 

Percent  of  Variance 

65 

21 

10 

Eigenvalues 

33.26 

9.15 

1.64 

Note  N = 42.  The  correlation  matrix  was  based  on  occupation-level  correlations  between  abilities  and 
generalized  work  activities.  FI  = High-Level  Cognitive,  Interpersonal,  and  Managerial  Skills,  F2  = 
Technical  Skills  with  Repair  Activities,  F3  = Programming.  These  loadings  are  based  on  an  orthogonal 
varimax  rotation. 


Table  13-8 

Principal  Components  Analysis  Pattern  Matrix  for  Cross-Domain  Analysis  of  Correlations  with 
Generalized  Work  Activities:  Knowledges 


Factor 


Knowledge  Descriptors 

FI 

F2 

F3 

F4 

Communality 

1.  Administration  and  Management 

.90 

.30 

-.14 

-.17 

.95 

2.  Clerical 

.78 

.08 

-.43 

-.08 

.81 

3.  Economics  and  Accounting 

.75 

.34 

-.22 

-.42 

.90 

4.  Sales  and  Marketing 

.68 

.21 

-.16 

-.59 

.87 

5.  Customer  and  Personal  Service 

.64 

.48 

-.37 

-.33 

.88 

6.  Personnel  and  Human  Resources 

.80 

.48 

-.18 

-.10 

.91 

7.  Production  and  Processing 

-.25 

-.03 

.84 

.12 

.79 

8.  Food  Production 

-.70 

-.02 

.60 

.20 

.87 

9.  Computers  and  Electronics 

.93 

-.06 

-.27 

.01 

.94 

10.  Engineering  and  Technology 

-.02 

-.34 

.78 

.47 

.95 

11.  Design 

.58 

-.07 

.73 

.11 

.89 

12.  Building  and  Construction 

-.40 

.00 

.90 

.10 

.98 

13.  Mechanical 

-.39 

-.19 

.78 

.43 

.98 

14.  Mathematics 

.96 

.05 

.00 

.13 

.95 

15.  Physics 

-.27 

.08 

.79 

.53 

.98 

16.  Chemistry 

-.28 

.04 

.56 

.76 

.98 

17.  Biology 

.16 

.14 

.16 

.94 

.96 

18.  Psychology 

.60 

.70 

-.36 

.06 

.99 

19.  Sociology  and  Anthropology 

.57 

.77 

-.24 

.00 

.97 

20.  Geography 

.48 

.83 

-.01 

-.10 

.93 

21.  Medicine  and  Dentistry 

-.10 

.45 

.22 

.84 

.97 

22.  Therapy  and  Counseling 

.48 

.83 

-.11 

.08 

.93 

23.  Education  and  Training 

.80 

.51 

-.15 

.16 

.96 

24.  English  Language 

.87 

.39 

-.28 

-.09 

.99 

25.  Foreign  Language 

-.23 

.87 

.36 

.05 

.94 

26.  Fine  Arts 

.39 

.72 

.30 

.12 

.78 

27.  History  and  Archeology 

.50 

.79 

.06 

.19 

.92 

28.  Philosophy  and  Theology 

.47 

.86 

-.18 

-.04 

.99 

29.  Public  Safety  and  Security 

-.27 

.86 

-.05 

.15 

.84 

30.  Law,  Government,  and  Jurisprudence 

.42 

.78 

-.38 

.09 

.95 

31.  Telecommunications 

.71 

.53 

-.04 

-.20 

.84 

32.  Communications  and  Media 

.81 

.52 

-.27 

-.07 

.84 

33.  Transportation 

-.33 

.48 

.62 

-.23 

.77 

Percent  of  Variance 

35 

27 

19 

12 

Eigenvalues 

17.43 

7.31 

3.43 

2.17 

Note.  N = 42.  The  correlation  matrix  was  based  on  occupation-level  correlations  between  knowledges 
and  generalized  work  activities.  FI  = Management/Business  and  Professional,  F2  = Social  Sciences  and 
Humanities,  F3  = Technical,  F4  = Science  and  Medicine.  These  loadings  are  based  on  an  orthogonal 
varimax  rotation. 
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Table  13-9 

Principal  Components  Analysis  Pattern  Matrix  for  Cross-Domain  Analysis  of  Correlations  with 
Generalized  Work  Activities:  Work  Context 

: Factor 


Work  Context  Descriptors 

FI 

F2 

F3 

F4 

Communality 

1. 

Communication  Formality 

-.56 

.69 

.30 

.29 

.97 

2. 

Communication  Methods 
2a.  Face-to-Face  Individuals 

-.09 

.05 

.64 

.15 

.44 

2b.  Face-to-Face  Groups 

-.33 

.82 

.23 

-.1 

.83 

2c.  Public  Speaking 

-.34' 

.26 

.81 

-.23 

.89 

2d.  Video  Conference 

-.52 

.36 

.52 

-.12' 

.68 

2e.  Voice  Mail 

-.80 

.48 

.25 

.04 

.93 

2f.  Telephone 

-.78 

.27 

.39 

.26 

.90 

2g.  Interactive  Computer 

-.72 

.10 

.58 

.16 

.89 

2h.  Electronic  Mail 

-.89 

.41 

-.02 

.07 

.97 

2i.  Handwritten  Notes 

-.67 

.43 

.44 

.27 

.90 

2j.  Letters  and  Memos 

-.74 

.51 

.31 

.01 

.91 

2k.  Written  Reports 

.10 

.30 

.17 

.88 

.90 

3. 

Communication  Subjectivity 

.03 

.59 

.59 

-.37 

.84 

4. 

Social  Interaction 

-.17 

.25 

.91 

.12 

.94 

5. 

Privacy  of  Communication 

-.74 

.61 

.14 

.06 

.94 

6. 

Job  Interactions 

6a.  Supervise/Develop  Others 

.15 

.79 

.02 

.02 

.65 

6b.  Persuade  or  Influence 

-.66 

.53 

.37 

-.25 

.92 

6c.  Provide  Service  to  Others 

-.28 

.12 

.73 

-.05 

.62 

6d.  Take  Opposing  Position 

-.50 

.78 

-.19 

-.09 

.90 

6e.  Work/Contribute  to  Team 

-.06 

.50 

.13 

-.07 

.27 

6f.  Deal  with  Public 

-.42 

.19 

.85 

.05 

.93 

6g.  Coordinate/Lead  Activity 

-.08 

.91 

.17 

-.05 

.87 

7. 

Health/Safety  of  Others 

.98 

.04 

.01 

.03 

.96 

8. 

Responsible  Others  Work 

.05 

.94 

.03 

-.17 

.91 

9. 

Conflict  Situations 

-.13 

.71 

.58 

.18 

.89 

10. 

Unpleasant  Individuals 

.29 

.07 

.86 

.27 

.90 

11. 

Physical  Aggression 

.66 

.18 

.47 

.48 

.92 

12. 

Work  Settings 

12a.  Indoors,  Controlled 

-.92 

.11 

.25 

-.06 

.92 

12b.  Indoors,  Uncontrolled 

.83 

-.08 

-.29 

-.19 

.81 

12c.  Outdoors,  Exposed 

.93 

-.04 

.08 

.02 

.88 

12d.  Outdoors,  Covered 

.91 

-.15 

.04 

-.10 

.86 

12e.  Open  Vehicle/Equipment 

.88 

-.13 

-.32 

.00 

.90 

12f.  Enclosed  Vehicle/Equipment 

.70 

.16 

.23 

.13 

.59 

13. 

Privacy  of  Work  Area 

-.73 

.64 

.19 

.05 

.98 

14. 

Physical  Proximity 

.82 

-.14 

.21 

.25 

.79 

15. 

Environmental  Conditions 
15a.  Distracting  Noise 

.93 

-.11 

-.02 

-.03 

.87 

15b.  Extreme  Temperatures 

.94 

-.20 

-.19 

-.07 

.96 

15c.  Poor  Lighting 

.93 

-.24 

-.13 

.06 

.94 

15d.  Contaminants 

.95 

-.23 

-.14 

.04 

.97 

Table  13-9  (continued) 

Principal  Components  Analysis  Pattern  Matrix  for  Cross-Domain  Analysis  of  Correlations  with 
Generalized  Work  Activities:  Work  Context 

Factor 


Work  Context  Descriptors 

FI 

F2 

F3 

F4 

Communality 

15e.  Cramped  Work  Space 

.92 

-.31 

-.19 

.02 

.98 

15f.  Whole  Body  Vibration 

.88 

-.18 

-.29 

-.05 

.87 

16a.  Radiation:  Likelihood  of  Injury 

.46 

-.06 

.39 

.61 

.73 

17.  Diseases/Infections 

.16 

.05 

-.71 

.14 

.55 

18.  High  Places 

.89 

-.21 

-.32 

-.16 

.97 

19.  Hazardous  Conditions 

.87 

-.18 

-.42 

.13 

.98 

20.  Hazardous  Equipment 

.92 

-.15 

-.31 

-.02 

.97 

21.  Hazardous  Situations 

.94 

-.25 

-.21 

-.05 

.98 

22.  Body  Positioning 

22a.  Sitting 

-.82 

.38 

-.01 

.32 

.92 

22b.  Standing 

.80 

-.40 

.01 

-.29 

.89 

22c.  Climbing  Ladders,  etc. 

.85 

-.24 

-.39 

-.21 

.97 

22d.  Walking  or  Running 

.85 

-.36 

.05 

-.31 

.95 

22e.  Kneeling  or  Crouching 

.86 

-.40 

-.11 

-.20 

.95 

22f.  Keeping/Regaining  Balance 

.93 

-.25 

-.09 

-.17 

.96 

22g.  Handling  Tools,  Objects 

.77 

-.56 

-.29 

-.10 

.99 

22h.  Bending/Twisting  Body 

.86 

-.45 

-.01 

-.20  ' 

.98 

22i.  Making  Repetitive  Motions 

.60 

-.69 

-.35 

-.05 

.97 

23.  Work  Attire 

23a.  Business/Office  Clothes 

-.82 

.34 

.43 

.06 

.98 

23b.  Special  Uniform 

.67 

-.27 

.32 

.43 

.81 

23c.  Maintenance  Clothes 

.82 

-.29 

-.44 

-.20 

.99 

23d.  Common  Safety  Attire 

.90 

-.19 

-.33 

.16 

.98 

23e.  Special  Safety  Attire 

.93 

-.11 

-.19 

.11 

.93  -- 

24f . Consequence  of  Error 

.48 

.49 

-.34 

.61 

.95 

25  .Level  of  Decisions 

25a.  Impact  of  Decisions 

-.31 

.76 

.33 

.42 

.96 

25b.  Frequency  of  Decisions 

-.19 

.58 

.67 

.36 

.95 

26.  Accountable  for  Results 

-.31 

.85 

.11 

.28 

.90 

27.  Decision  Latitude 

-.46 

.81 

.25 

.18 

.98 

28.  Frustrating  Circumstances 

-.45 

.72 

-.30 

.26 

.87 

29.  Level  of  Automation 

-.71 

-.07 

-.31 

.43 

.79 

30.  Task/Performance  Clarity 

.63 

-.62 

.15 

.16 

.83 

31.  Accuracy/Exactness 

-.54 

-.03 

.18 

.78 

.93 

32.  Details  and  Completeness 

-.50 

-.15 

.03 

.71 

.78 

33.  Constant  Awareness 

.72 

.00 

.55 

.34 

.94 

34.  Repetitive  Activities 

.58 

-.73 

-.15 

.25 

.95 

35.  Unstructured  Tasks/Goals 

-.62 

.72 

.21 

-.07 

.95 

36.  Level  of  Competition 

.09 

.41 

.26 

. -.64 

.65 

37.  Deadlines/Time  Pressure 

.88 

-.16 

. -.27 

.18 

.90 

38.  Work  With  Distractions 

-.64 

.39 

.18 

.50 

.85 

39.  Machine  Driven  Pace 

.63 

-.50 

-.50 

.03 

.90 

Table  13-9  (continued) 

Principal  Components  Analysis  Pattern  Matrix  for  Cross-Domain  Analysis  of  Correlations  with 
Generalized  Work  Activities:  Work  Context 

Factor 

Work  Context  Descriptors  FI  F2  F3  F4  Communality 

Percent  of  Variance  47  20  14  8 

Eigenvalues 45.72  10.67 &61 5.56 

Note.  N = 42.  The  correlation  matrix  was  based  on  occupation-level  correlations  between  abilities  and 
generalized  work  activities.  FI  = Hazardous  Work  Environments,  F2  = Managerial  Relations,  F3  = 
Interpersonal:  Especially  Public  and/or  Conflict,  F4  = Careful  Documentation.  These  loadings  are  based 
on  an  orthogonal  varimax  rotation. 
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Chapter  14 

Occupation  Classification:  Using  Basic  and  Cross-Functional 
Skills  and  Generalized  Work  Activities  to  Create  Job  Families 

Dwayne  G.  Norris 
Wayne  A.  Baughman 
Ashley  E.  Cooke 
Norman  G.  Peterson 
Michael  D.  Mumford 
American  Institutes  for  Research 


This  chapter  presents  results  of  job  clustering  analyses  using  the  measures  of  broad  skills 
and  work  activities  from  the  0*NET  content  model.  These  measures  consist  of  46  basic  and 
cross-functional  skills  and  42  GW  As.  We  begin  with  a brief  introduction  describing  the  specific 
objectives  of  the  job  clustering  analyses,  a description  of  the  data  used,  and  background  on  the 
analytic  approach.  We  then  present  and  discuss  the  results  and  their  implications  for  occupation 
classification  using  the  0*NET  taxonomy. 

Introduction 

The  overall  objective  of  the  job  clustering  analyses  was  to  answer  the  following  question: 
What  is  the  utility  of  the  newly  developed  0*NET  measures  of  basic  and  cross-functional  skills 
and  generalized  work  activities  (GW As)  for  job  classification  purposes?  This  is  a central 
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question  because  these  descriptors  were  designed  specifically  for  comparing  occupations  within 
and  across  organizations.  Furthermore,  these  measures  of  skills  and  work  activities  represent  the 
two  domains  of  primary  focus  in  most  prior  classification  work,  namely,  human  attributes  of 
work  performance  and  the  nature  of  work  itself  (Pearlman,  1980).  Hence,  these  0*NET  measures 
represent  alternative  perspectives  commonly  used  to  describe  jobs. 

Occupation  classification  involves  grouping  jobs  or  occupations  on  the  basis  of  shared 
characteristics,  such  as  tasks  performed  or  skills  required.  In  personnel  psychology  and  human 
resource  management,  classifying  jobs  or  occupations  represents  the  necessary  first  step  in 
performing  a wide  range  of  personnel  functions.  For  example,  in  wage  and  salary  administration, 
it  is  legally  advisable  to  place  highly  related  jobs  in  the  same  salary  range.  Similarly,  highly 
related  jobs  may  use  the  same  selection  and  performance  appraisal  instruments,  thereby 
significantly  decreasing  the  development  and  administrative  burden  associated  with  these 
•functions.  In  short,  job  classification  uses  job  description  information  to  determine  which  jobs 
are  administratively  equivalent,  and  therefore  simplifies  most  major  functions  of  personnel 
administration  greatly.  Understanding  the  characteristics  of  the  0*NET  skills  and  work  activities 
descriptors  in  identifying  highly-related  occupations,  or  job  families,  is  an  important  issue 
because  these  descriptors  are  intended  to  provide  descriptive  information  applicable  across  many 
jobs  and/or  occupations. 

Related  to  the  need  to  evaluate  the  classification  utility  of  the  0*NET  measures  of  skills 
and  work  activities  is  the  fact  that  the  Department  of  Labor  (DOL)  is  participating  in  the  national 
move  toward  a skills-based  emphasis  in  describing  jobs.  This  effort  stems  largely  from  the  ever 
increasing  global  economic  competition  which,  in  turn,  creates  the  need  for  continued  mastery  of 
new  skills  and  technologies  by  the  nation’s  workforce.  The  world  of  work  is  moving  toward  a 
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state  characterized  by  the  need  for  a high  degree  of  employment  flexibility  and  technical  know- 
how among  workers  of  most  occupations.  The  0*NET  basic  and  cross-functional  skills  and 
GW  As  represent  this  perspective  of  work,  and  thus  provides  a sound  basis  for  skill-based 
occupation  classification. 

Analysis  Overview 

The  characteristics  of  the  basic  and  cross-functional  skills  and  GW  As  were  examined 
with  respect  to  three  issues  that  affect  the  results  of  any  classification  effort.  Specifically,  this 
study  compared  the  quality  of  cluster  solutions  which  were  based  on: 

• different  variable  sets, 

• different  clustering  procedures,  and 

• different  samples. 

Prior  research  demonstrates  that  job  description  variables,  clustering  procedures,  and  samples  can 
influence  the  results  of  clustering  efforts  (e.g.,  Cornelius,  Carron,  & Collins,  1979).  Therefore, 
these  three  areas  formed  critical  domains  across  which  cluster  solutions  were  examined. 

First,  cluster  solutions  based  on  different  variable  sets  were  compared.  As  noted  above, 
the  0*NET  job  descriptors  chosen  for  these  analyses  included  basic  and  cross-functional  skills 
and  GW  As.  Each  of  these  descriptors  are  defined  broadly  enough  to  make  comparisons  across 
jobs.  Thus,  this  first  comparison  helps  reveal  differences  between  job  cluster  solutions  using 
different  descriptors.  These  analyses  resulted  in  comparisons  of  job  family  solutions  based  on  (1) 
GW  As,  (2)  basic  and  cross-functional  skills,  or  (3)  GW  As  and  skills  combined. 

Second,  the  cluster  solutions  from  two  types  of  agglomerative  hierarchical  clustering 
methods  were  compared.  In  general,  cluster  analysis  is  a class  of  statistical  techniques  in  which 
cases  (e.g.,  occupations)  are  grouped  on  the  basis  of  their  similarities  across  a set  of  descriptive 
variables  (e.g.,  job  descriptors).  Agglomerative  hierarchical  methods  represent  one  of  several 
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approaches  to  conducting  a cluster  analysis.  While  other  clustering  methods  exists  (e.g.,  divisive 
hierarchical,  k-medoid),  agglomerative  hierarchical  clustering  is  widely  used  and  properties  of 
this  method  are  thoroughly  understood.  These  analyses  involved  a comparison  of  clustering 
solutions  based  on  two  widely  used  algorithms  for  developing  clusters  with  agglomerative 
hierarchical  procedures. 

Finally,  cluster  solutions  based  on  different  samples  were  compared.  Data  for  both  job 
incumbents  and  occupational  analysts  were  available  and  cluster  solutions  across  these  samples 
were  compared.  These  analyses  were  intended  to  reveal  whether  differences  between 
incumbents’  and  analysts’  ratings  would  produce  differences  in  cluster  solutions.  This 
comparison  has  obvious  implications  for  future  data  collection  efforts  using  the  0*NET 
taxonomy  because  analysts’  ratings  of  occupations  are  widely  used  for  a variety  of  purposes. 

These  comparisons  helped  evaluate  the  overall  quality  of  the  various  cluster  solutions 
without  regard  to  the  specific  assignment  of  occupations  to  clusters.  However,  an  important 
outcome  stemming  from  these  comparisons  was  a determination  of  whether  or  not  the  resulting 
cluster  solutions  are  qualitatively  similar  in  terms  of  occupation  assignment.  We  concluded  these 
analyses  by  assessing  the  extent  to  which  the  occupations  were  being  cluster  in  the  same  or 
different  clusters  across  the  various  conditions  noted  above. 

Data  Description 
Database  Construction 

Three  databases  were  available  for  this  investigation.  The  first  database,  called  the  “1,122 
analysts  data,  contained  ratings  made  by  trained  occupational  analysts.  These  occupational 
analysts  consisted  of  occupational  analysts  and  Industrial/Organizational  psychology  graduate 
students.  The  analysts  rated  1,122  occupational  units,  where  occupational  units  represented  a 
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taxonomy  of  jobs  with  a level  of  specificity  in  between  the  more  general  Occupational 
Employment  Statistics  (OES)  taxonomy  and  the  more  specific  Dictionary  of  Occupational  Titles 
(DOT)  taxonomy.  Ratings  occurred  on  a subset  of  the  0*NET  content  model  categories, 
including  basic  and  cross-functional  skills  and  GW  As.  At  any  one  time,  five  analysts 
independently  rated  the  level,  importance,  and/or  frequency  of  125  occupational  units  on  a single 
content  model  category  (e.g.,  GW  As).  Analysts  were  rotated  to  fit  this  data  collection  structure, 
thereby  ensuring  a mix  of  raters  and  minimizing  rating  errors.  From  completed  ratings,  mean 
level  scores  on  each  descriptor  served  as  the  rating  for  each  occupational  unit.  Thus,  the  1,122 
analysts  data  contained  mean  ratings  for  1,122  occupational  units  on  level,  importance,  and 
frequency,  if  applicable,  for  the  selected  0*NET  content  model  categories. 

The  second  database,  called  the  “29  incumbent  data,”  was  comprised  of  level, 
importance,  frequency,  and/or  job  entry  requirement  ratings  for  29  occupational  units  by 
approximately  700  incumbent  workers.  It  is  estimated  that  these  29  occupational  units  encompass 
34  percent  of  the  employed  population.  Also,  these  29  occupational  units  represent  a subset  from 
an  original  list  of  80  targeted  occupational  units  where  four  or  more  incumbents  responded  to  all 
of  the  0*NET  data  collection  questionnaires.  Job  incumbents  typically  completed  more  than  one 
and  as  many  as  five  questionnaires.  As  with  the  analysts  data,  mean  incumbents’  ratings  on  level, 
importance,  and  frequency,  if  applicable,  across  the  job  descriptors  served  as  the  occupational 
unit  scores.  Thus,  the  incumbent  database  contains  mean  ratings  on  29  occupational  units,  as 
opposed  to  the  1,122  rated  occupational  units  of  the  complete  analyst  database. 

Finally,  a third  database  consisting  of  29  occupational  units  corresponding  to  those 
occupations  rated  by  incumbent  workers  was  abstracted  from  the  1,122  analysts  data.  This 


14-6  Norris,  Baughman,  Cooke,  Peterson,  & Mumford 


database  is  called  the  “29  analysts  data.”  Because  this  extracted  database  is  a subset  of  the  1,122 
analysts  data,  it  shares  all  the  other  properties  of  that  larger  database. 

From  each  of  these  three  databases,  only  the  level  ratings  were  used  in  the  current 
analyses. 

Description  of  Variables 

To  this  point,  basic  and  cross-functional  skills  and  GW  As  have  been  described  only  as 
being  broadly  defined.  In  the  0*NET  framework,  basic  and  cross-functional  skills  represent 
general  procedures  or  strategies  contributing  to  performance  on  job  tasks.  Basic  and  cross- 
functional skills  are  regarded  as  two  key  kinds  of  capacities  in  performance.  First,  basic  skills 
represent  the  performance  capacities  of  individuals  which  facilitate  learning  or  the  more  rapid 
acquisition  of  knowledge  in  general,  whereas  cross-functional  skills  represent  experienced-based 
performance  capacities  and  contribute  to  performance  across  many  jobs.  Examples  of  the  basic 
skills  include  Reading  Comprehension  (Skill  Descriptor  #1)  and  Critical  Thinking  (Skill 
Descriptor  #7),  and  examples  of  cross-functional  skills  include  Information  Gathering  (Skill 
Descriptor  #18)  and  Solution  Appraisal  (Skill  Descriptor  #24). 

Generalized  work  activities  are  broadly  defined  work  functions.  As  such,  GW  As  are  like 
basic  and  cross-functional  skills  in  that  they  remain  equally  applicable  across  many  jobs  and 
occupations.  An  example  of  a GW  As  is  Interacting  with  Computers  (GWA  Descriptor  #19).  As  a 
work  function,  interacting  with  computers  likely  applies  to  many  seemingly  distinct  occupations 
(e.g.,  computer  analysts,  secretary)  and  thus  provides  a mechanism  for  describing  and  classifying 
occupations. 
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Clustering  Technique 

There  are  many  decisions  to  make  when  conducting  a cluster  analysis.  Prior  research 
provides  some  basic  guidelines,  and  this  analysis  used  these  guidelines  in  an  effort  to  focus  the 
scope  of  the  analyses  and  eliminate  potential  methodological  confounds. 

In  terms  of  the  analytic  scope,  these  analyses  examined  the  characteristics  of  the  basic  and 
cross-functional  skills  and.GWAs  with  respect  to  two  clustering  algorithms  and  two  modes  of 
combining  descriptors  across  three  samples.  Where  possible,  decisions  regarding  specific  aspects 
of  clustering  methodology  not  directly  relevant  to  exploring  these  issues  were  standardized.  All 
standardization  decisions  were  made  on  the  basis  of  supported  recommendations  from  the 
available  literature.  Once  made,  these  aspects  of  the  analyses  remained  constant.  For  example. 
Euclidean  distance  was  chosen  as  the  distance  measure,  and  thus  all  the  analyses  used  this 
measure  without  exception. 

The  second  advantage  to  this  standardization  approach  was  that  it  reduced  the  influence 
of  potential  methodological  confounds.  It  would  be  difficult  to  interpret  differences  among 
cluster  solutions  if  the  methods  used  to  produce  those  solutions  varied  substantially.  That  is,  the 
more  elements  of  the  research  design  that  could  be  fixed,  the  better. 

Clustering  Approach 

There  were  essentially  four  major  analytic  decisions  to  make  in  conducting  these 
analyses.  These  decisions  concerned  data  standardization,  profile  similarity  measurement, 
clustering  methodology,  and  criteria  for  evaluating  cluster  solutions.  Each  of  the  decisions  are 
discussed  below.  First,  however,  it  is  important  to  understand  another  aspect  of  cluster  analysis, 
data  conversion,  which  pertains  to  how  the  data  are  treated  and  not  what  they  are  or  where  they 
originate  (these  latter  issues  have  been  previously  addressed). 
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Most  often,  data  consist  of  individual  ratings  on  a series  of  variables  pertaining  to  some 
domain(s)  of  interest.  Here,  however,  data  consisted  of  mean  occupational  ratings  on  several  job 
descriptors.  The  level  of  analysis  is  the  occupation  and  not  the  individual.  Furthermore, 
occupations  form  the  targeted  domain  in  the  data  analysis;  the  relationship  among  occupations 
across  variables  is  the  focus,  as  opposed  to  the  relationship  of  variables  across  occupations. 

Data  Standardization 

These  analyses  clustered  occupational  groupings  with  respect  to  the  shape  of  their  profile 
only,  where  profile  refers  to  the  overall  pattern  of  responses  across  a set  of  job  descriptors.  Both 
the  scatter  and  level  dimensions  of  occupational  grouping  profiles  were  eliminated  by 
standardizing  the  data  within  each  occupational  grouping.  This  required  converting  the  skills  and 
work  activities  scales,  separately,  to  z-score  values  within  each  occupation.  The  effect  of  this 
procedure  was  to  give  each  occupational  grouping,  or  case,  a mean  value  of  0 and  a standard 
deviation  of  1 across  each  set  of  skill  and  work  activities  measures. 

The  decision  to  create  occupational  families  using  only  the  shape  dimension  of  the  score 
profile  centered  on  three  reasons.  First,  shape  represents  a primarily  qualitative— as  opposed  to  a 
quantitative— dimension  of  a profile.  It  can  be  argued  that  the  most  coherent  description  of 
occupational  families  is  one  that  emphasizes  qualitative  differences:  differences  in  kind  or  type. 
Thus,  occupational  groupings  that  share  the  same  relative  emphasis  on  skills  and  work  activities 
get  grouped  together,  ignoring  differences  between  occupations  in  the  level  or  variability  of  these 
skills  or  work  activities. 

Clustering  on  profile  measures  that  take  into  account  (or  even  emphasized)  the  level,  or 
amount,  of  these  attributes,  might  result  in  occupations  being  assigned  to  separate  clusters  in 
ways  that  would  be  difficult  to  explain.  For  example,  it  is  typical  for  occupations  in  the  same 
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career  field  to  differ  primarily  by  level  of  skill  required  (e.g.,  entry-,  mid-,  and  executive-level). 
Here,  the  career  field  represents  qualitatively  similar  occupations  because  they  all  share  a similar 
shape  of  skill  profile,  but  differ  with  respect  to  the  level  of  skills  required.  If  differences  in  skill 
level  dominated  the  construction  of  occupational  families,  then  some  occupations  may  be 
grouped  with  other  occupations  from  different  career  fields  because  they  require  the  same  overall 
level  of  different  skills.  Such  clustering  makes  the  subsequent  occupational  structure  difficult  to 
interpret  and  use. 

The  second  reason  for  clustering  on  shape  is  that  level  differences  within  a job  family  can 
be  relatively  straightforwardly  identified  post-hoc  by  looking  at  mean  score  differences  across 
profile  variables.  Furthermore,  there  are  clear  advantages  to  looking  at  level  after  occupational 
groupings  have  been  constructed  based  on  the  shape  of  the  profile.  For  example,  such  a post-hoc 
identification  of  level  differences  allows  the  identification  of  career  paths,  career  development 
plans,  promotion  tracks,  and  training  needs.  Hence,  level  effects  are  not  regarded  as  meaningless; 
indeed  they  are  extremely  useful,  but  are  simply  identified  at  a later  phase  of  the  analysis.  The 
same  holds  true  for  score  profile  variability,  or  scatter  effects.  After  job  family  construction, 
differences  within  family  with  respect  to  profile  score  scatter  can  be  identified  by  looking  at 
variability  across  descriptor  measures. 

Finally,  clustering  on  shape  reduces  the  level  of  complexity  in  the  distance  measure  by 
having  it  reflect  a single  profile  dimension.  This  should  enhance  the  interpretability  of  the 
resulting  occupational  structure,  and  thus  reduce  the  likelihood  of  ambiguous  interpretations  that 
may  arise  from  solutions  representing  information  about  shape,  level,  and  scatter  simultaneously. 
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Distance  Measure 

In  cluster  analysis,  grouping  cases  proceeds  on  the  basis  of  commonality,  as  defined  by  a 
dissimilarity  or  similarity  measure.  Euclidean  distance,  or  D,  represented  the  measure  of  distance 
between  occupational  groupings  for  these  analyses.  Euclidean  distance  between  two  items,  X and 
Y,  is  the  square  root  of  the  sum  of  squared  differences  between  all  values  for  those  items 
(Cronbach  & Gleser,  1953;  Norusis/SPSS,  Inc.,  1993).  With  raw  score  data,  D represents  all 
three  dimensions  of  a profile  score:  scatter,  level,  and  shape.  However,  given  the  within-case 
standardization  of  the  data,  both  scatter  and  level  dimensions  were  removed. 

Euclidean  distance  is  used  extensively  in  clustering  applications  and  therefore  is  well 
understood  in  terms  of  its  properties.  Unlike  Squared  Euclidean  distance.  Euclidean  distance 
does  not  exaggerate  larger  distances  between  objects  (Cronbach  & Gleser,  1953).  Further,  use  of 
Euclidean  distances  is  currently  considered  the  preferred  distance  metric  in  occupational 
clustering  (Colihan  & Burger,  1995)  and  for  most  other  applications  (Kaufmann  & Rousseeuw, 
1990). 

Methodology 

Agglomerative  hierarchical  clustering  algorithms  were  used  for  developing  the 
occupational  groupings.  All  agglomerative  hierarchical  procedures  begin  by  treating  each  case  as 
a distinct  group,  then  iteratively  building  increasingly  larger  clusters  until  all  cases  are  clustered 
together.  These  procedures  first  identify  the  two  cases  with  the  most  similar  profiles  on  the 
clustering  variables  based  on  a procedure-specific  criterion.  Next,  the  remaining  cases  are 
evaluated  and  either  used  to  form  a new,  different  cluster,  or  assigned  to  an  existing  cluster.  As 
the  cases  are  combined,  the  metric  representing  the  distance  between  groups  is  recalculated.  The 
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procedure  ends  when  all  the  cases  are  combined  into  one  cluster.  It  is  clear  from  this  description 
that  hierarchical  procedures  produce  groups  of  increasing  levels  of  generality. 

While  this  is  the  overall  process  for  all  agglomerative  procedures,  there  exists  many 
different  clustering  methods  for  combining  the  occupational  units  into  clusters.  We  used  the 
Ward  and  Hook  (“Ward’s”  method;  Ward  & Hook,  1963)  and  the  within-groups  average  linkage 
methods.  These  two  methods  were  used  for  two  reasons.  First,  for  both  methods,  the 
mathematical  model  represented  by  the  algorithms  reflects  the  nature  of  the  intended  application 
(Everitt,  1993).  Both  Ward's  and  within-group  average  linkage  methods  attempt  to  find  clustering 
solutions  that  maximize  within-group  similarity.  The  Ward's  method  seeks  to  minimize  the 
within-groups  error  (and  thus  maximize  between-groups  variance),  while  the  within-group 
average  linkage  method  attempts  to  minimize  within-group  distance  from  the  group  ceijtroid.  In 
the  present  effort,  the  result  would  be  an  occupational  structure  in  which  occupations  within  each 
cluster  would  have  the  most  similar  profiles  of  skills  and  work  activities.  The  occupational 
families  themselves,  however,  would  tend  to  have  dissimilar  average  profiles  on  the  skills  and 
work  activities  scales.  Clearly,  these  models  are  consistent  with  the  objectives  of  typical 
occupational  family  development  efforts. 

Second,  both  the  Ward's  within-group  average  linkage  methods  have  track  records  of 
reliable  application  both  generally  and  in  developing  clusters  of  people  or  jobs  (Colihan  & 
Burger,  1995).  For  example,  Everitt  (1993)  recommends  Ward's  and  group  average  methods  as 
the  most  widely  applicable  of  the  hierarchical  class  of  techniques  (p.  142).  Ward's  method  has 
been  widely  adopted  by  researchers  in  areas  as  diverse  as  botany,  archeology,  and  sociology 
(Kaufmann  & Rousseeuw,  1990).  Furthermore,  Ward's  method  is  considered  by  those  working  in 
the  area  of  occupation  classification  to  be  one  of  the  most  commonly  used  and  effective  of  the 
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available  clustering  methods  (Harvey,  1986;  1990).  Average  linkage  methods  were  shown  in  a 
recent  Monte  Carlo  study  to  recover  clusters  as  well  as  Ward's  method  when  combined  with 
other  methods  for  defining  the  number  of  clusters  in  the  data  (Colihan  & Burger,  1995). 

Use  of  Job  Descriptors 

There  are  many  possible  ways  to  use  GW  As  and  basic  and  cross-functional  skills  as  job 
descriptor  information,  and  the  present  investigation  explored  five  specific  ways  for  using  the  job 
descriptor  information.  Generally  speaking,  these  approaches  involved  using  the  skills  and  GWA 
data  either  separately  or  in  combination  to  form  job  clusters. 

The  first  two  approaches  evaluated  the  utility  of  using  GW  As  or  basic  and  cross- 
functional  skills  to  the  exclusion  of  the  other  category  of  job  descriptor  information.  This  method 
requires  less  data  and  thus  represents  a more  parsimonious  approach  from  a data  requirement 
perspective.  However,  the  utility  of  this  approach  depends  on  many  other  factors,  including  the 
extent  to  which  job  families  based  on  either  one  are  useful  and  clearly  defined  given  the 
exclusion  of  the  other  category  of  job  descriptor  data. 

The  remaining  three  approaches  under  investigation  involved  using  both  types  of 
descriptor  information  together  in  deriving  the  job  families.  The  more  straight-forward  approach 
involved  using  all  GWA  and  skills  descriptors  at  a single  clustering  step.  Also,  two-stage 
clustering  methodologies  were  employed.  In  these  approaches,  the  occupations  were  clustered 
first  on  one  set  of  descriptors  (i.e.,  skills  or  GW  As).  Then,  separate  cluster  analyses  were  carried 
out  within  each  initial  cluster,  but  using  the  alternate  set  of  descriptors. 

The  two-stage,  hierarchical  approach  to  clustering  provides  a unique  approach  to  job 
classification  in  which  the  type  of  descriptor  data  represents  different  domains.  There  are  several 
reasons  for  trying  out  such  an  approach.  It  is  difficult  to  know,  a priori,  whether  or  not  one  type 
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of  data  (e.g.,  GW  As)  will  dominate  the  classification  process  if  several  types  of  data  are  clustered 
simultaneously.  The  two-staged  approach  allows  for  a more  controlled  usage  of  the  job 
descriptors.  Additionally,  this  approach  might  be  useful  if  there  is  some  rationale  for  a particular 
order  of  clustering.  For  instance,  it  might  be  argued  that  clustering  first  on  skills  produces  more 
stable  job  families  given  the  more  enduring  nature  of  skills  in  comparison  to  work  activities. 
Then,  clustering  on  GW  As  within  each  skills-based  job  family  demonstrates  how  similar  sets  of 
skills  may  be  applied  to  different  work  activities.  This  approach  might  represent  one  perspective 
of  work  in  which  employees  first  master  skills  (e.g.,  college  education)  before  applying  those 
skills  to  specific  domains  (e.g.,  on  the  job). 

We  applied  the  two-stage  approach  in  two  ways:  using  skills,  then  GW  As,  and  vice-versa. 

Criteria 

This  section  describes  a set  of  objective  criteria  used  to  evaluate  the  quality  of  the  cluster 
solutions  obtained  from  our  analyses.  These  objective  criteria  are  ways  of  representing  cluster 
solution  stability  and  coherence.  Of  course,  in  practice,  subjective  criteria  (e.g.,  apparent  utility, 
acceptability,  or  parsimony  for  particular  purposes)  are  equally  important  in  evaluating  particular 
cluster  solutions.  However,  these  analyses  emphasized  more  objective  criteria  to  better  facilitate 
making  multiple  comparisons  across  multiple  clustering  solutions.  The  specific  objective  criteria 
are:  percent  of  cases  correctly  reclassified,  simple  structure,  mean  squared  error,  and  effect  size. 

Percent  for  Number)  Correctly  Reclassified 

The  percent  of  respondents  correctly  reclassified  provides  an  index  of  the  relative  stability 
of  the  initial  assignments  of  cases  to  clusters  using  a “confirmatory  k-means”  procedure. 
Operationally,  the  centroids  (i.e.,  group  means)  for  each  descriptor  are  obtained  from  the  initial 
hierarchical  procedure.  These  centroids  are  used  to  define  initial  target  cluster  centers  for  a k- 
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means  clustering,  or  partitioning,  procedure.  The  k-means  procedure  then  assigns  occupations  to 
clusters  based  on  these  initial  cluster  centers.  Finally,  the  initial  hierarchical  solution  and  the  k- 
means  solution  are  compared  to  identify  the  number  of  occupations  that  are  classified  differently. 
Given  a good  initial  cluster  solution,  this  test  should  find  little  or  no  change  between  cluster 
solutions. 

A k-means  partitioning  analysis  is  a cluster  analysis  that  is  neither  hierarchical  nor 
agglomerative.  Instead,  a k-means  procedure  begins  by  identifying  the  k most  diverse  cases  based 
on  the  number  of  clusters,  k,  specified  a priori.  This  first  set  of  k cases  provides  the  initial  cluster 
centers  to  which  the  other  cases  are  subsequently  assigned.  In  the  variation  used  here 
(MacQueen,  1967),  group  centroids  from  the  hierarchical  agglomerative  clustering  analysis  are 
used  as  the  cluster  centers  for  the  partitioning  analysis.  Then,  the  partitioning  analysis  will 
attempt  to  assign  the  occupations  to  groups  based  on  using  the  target  cluster  centers.  This  usually 
results  in  most  cases  being  reassigned  back  into  their  original  groups,  although  some  cases  are 
likely  to  be  assigned  to  groups  other  than  their  original  one.  Typically,  the  partitioning  analysis  is 
run  several  times  until  the  number  of  occupations  assigned  to  each  occupational  family  stabilizes, 
a process  known  as  “chaining.” 

Simple  Structure 

The  simple  structure  provides  an  index  of  the  relative  “fuzziness”  or  clarity  of  the 
assignment  of  occupations  to  clusters.  Specifically,  this  index  represents  the  proportion  of  final 
clusters  where  75%  or  more  of  the  occupations  in  each  cluster  have  at  least  a .75  probability  of 
being  a member  of  its  assigned  cluster.  Operationally,  these  probabilities  are  derived  from  a 
discriminant  classification  analysis.  The  procedure  develops  optimal  linear  composites,  or 
classification  functions,  for  each  cluster  using  the  set  of  descriptors,  and  then  creates  scores  on 
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the  classification  functions  for  each  occupation.  Occupations  are  then  assigned  to  clusters  based 
on  whichever  of  the  functions  yields  the  largest  value.  These  function  scores  are  also  used  to 
derive  the  probabilities  of  cluster  membership  for  each  occupation  that  are  used  to  form  this 
simple  structure  index. 

Mean  Squared  Error 

The  mean  squared  error  (MSE)  provides  an  index  of  the  average  homogeneity  of  the 
clusters,  comprising  each  solution,  disregarding  the  separation  between  clusters.  Having 
homogeneous  occupations  represented  in  the  same  job  family  is  the  primary  objective  of  most 
job  classification  efforts.  Most  often,  for  example,  we  are  interested  in  knowing  which 
occupations  we  may  treat  the  same  with  respect  to  personnel  administration,  such  as  developing 
training  programs  and  salary  administration.  Operationally,  this  index  is  the  error  term  derived 
from  a mixed-design  MANOVA  where  the  set  of  descriptors  serve  as  a set  of  repeated  measures 
and  the  occupational  clusters  serve  as  the  independent  variables.  The  error  term  from  this 
MANOVA  represents  the  average  variability  within  the  occupational  clusters. 

Effect  Size 

The  effect  size  provides  an  index  of  the  amount,  or  size,  of  the  separation  between  the 
occupational  clusters.  This  index  is  derived  from  the  same  MANOVA  that  provides  the  value  for 
Mean  Squared  Error,  and  represents  the  proportion  of  variance  between  the  clusters  relative  to 
the  total  variance. 

Table  14-1  presents  the  study  design  depicting  the  thirty  analyses  conducted  in  this 
investigation.  The  first  column  of  Table  14-1  contains  five  different  ways  that  the  two  descriptor 
sets  might  be  combined.  The  remaining  columns  display  the  three  samples,  each  with  the  Ward’s 
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and  average  linkage  clustering  methods  nested  under  them.  Cell  entries  include  the  number  of 
obtained  clusters,  number  of  single  occupation  clusters,  and  the  values  for  the  four  criteria. 

Analysis  Procedures 

This  section  briefly  outlines  the  analytic  process  followed  in  addressing  the  research 
questions.  For  each  of  these  steps,  particular  attention  is  given  to  the  information  provided  at  that 
step.  The  specific  procedures  will  not  be  described  in  detail  as  they  can  be  found  elsewhere  by 
the  interested  reader  (e.g.,  Everitt,  1993;  Harvey,  1986;  Pearlman,  1980). 

Standardized  Data 

The  first  step  of  each  analysis  was  to  create  two  matrices.  The  first  matrix  was  a 
dissimilarity  matrix  of  distances  based  on  the  z-score  transformation  of  the  skills  and  work 
activities  scores  described  earlier.  This  matrix  was  used  for  carrying  out  cluster  analysis.  The 
second  matrix  was  a correlation  matrix  representing  the  correlations  among  the  occupations 
across  the  skills  and  work  activities  scores.  This  matrix  was  used  to  cany  out  the  q-factor 
analyses  described  in  the  next  section. 

Determine  Number  of  Clusters 

The  number  of  clusters  to  extract  was  determined  in  two  ways.  For  the  1,122  analysts 
data,  number  of  clusters  was  determined  by  conducting  a q-factor  analysis  on  the  1,122  x 1,122 
correlation  matrix  of  occupations  created  during  data  standardization.  Q-factor  analysis 
represents  a class  of  principal  components  or  factor  analyses  in  which  the  cases  (i.e., 
occupational  units)  are  factored,  as  opposed  to  the  variables.  With  q-factor  analysis,  the  number 
of  potential  groups  in  the  data  can  be  identified  prior  to  clustering  using  the  same  factor 
identification  methods  of  traditional  factor  analyses.  Here,  the  eigenvalue  greater  than  one  rule 
(see  Gorsuch,  1 983)  served  as  the  guideline  for  setting  the  number  of  clusters  to  extract.  Recent 
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research  supports  the  viability  of  this  overall  approach  for  determining  number  of  clusters.  For 
example,  Monte  Carlo  results  suggests  that  the  use  of  cj-factor  analysis  to  identify  the  number  of 
groups  helps  to  improve  classification  results,  particularly  using  Ward’s  method  (Colihan  & 
Burger,  1995). 

For  the  29  analysts  data  and  29  incumbent  data,  the  number  of  clusters  was  determined  by 
examining  the  dendrograms.  Dendrograms  are  “tree-structured”  graphical  representations  that 
display  the  formation  of  clusters  at  each  stage.  They  may  be  displayed  together  with  rescaled 
clustering  coefficients  that  represent  the  distance  between  clusters  that  are  combined  at  each 
stage.  Hence,  large  increases  in  clustering  coefficients  indicate  that  very  dissimilar  elements 
and/or  clusters  have  been  formed.  Determining  the  number  of  clusters  proceeds  by  examining  the 
dendrogram  for  these  large  increases,  represented  by  longer  distances  in  the  tree-structured 
diagram. 

Hierarchical  Agglomerative  Clustering 

Once  the  number  of  clusters  was  determined,  clustering  analyses  were  conducted  for  each 
of  the  30  analyses  comprising  the  cells  of  Table  14-1.  Each  analysis  provided  initial  clusters,  as 
well  as  “seed  points”  (e.g.,  cluster  centroids)  for  the  next  step. 

K-Means  Clustering 

After  identifying  the  initial  clusters,  the  k-means  procedure  reclassified  all  occupations 
into  their  optimal  occupational  family.  This  procedure  is  often  necessary  because  hierarchical 
agglomerative  clustering  procedures  suffer  from  the  flaw  that  individual  cases  clustered  early  in 
the  procedure  may  not  end  up  being  optimally  classified.  Information  used  to  calculate  the 
percentage,  or  alternatively,  the  number  of  cases  correctly  reclassified  comes  from  comparing  the 
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initial  hierarchical  cluster  solution  to  the  reclassified  k-means  cluster  solutions  by  cross- 
tabulation. 

Discriminant/M  ANOVA  Analyses 

Finally,  the  resulting  clusters  from  the  k-means  were  used  in  discriminant  and  MANOVA 
analyses  to  obtain  additional  indices  for  evaluating  the  clustering  results.  Specifically,  this 

process  provides  indicators  for  simple  structure  of  the  cluster  solution,  effect  size,  and  mean 
squared  error. 

Results/Discussion 

The  overall  purpose  of  this  analysis  was  to  examine  the  utility  of  the  new  cross  job 
descriptors  from  the  0*NET  content  model  in  creating  job  families.  The  measures  of  basic  and 
cross  functional  skills,  and  GW  As  served  as  these  job  descriptors.  As  noted  earlier,  these  broad 
skills  and  work  activities  measures  were  appropriate  for  two  reasons.  First,  these  measure 
encompass  those  aspects  of  job  performance-- worker  attributes  and  job  duties—most  often  used 
for  the  creation  of  job  families  (Pearlman,  1980).  The  skills  measure  worker  attributes  and  the 
GW  As  measure  types  of  job  functions  and/or  duties.  Second,  these  measures  were  designed 
specifically  for  making  comparisons  across  jobs  and  occupations. 

Table  14-1  presents  the  results  of  this  study  in  terms  of  the  quality  of  the  cluster  solutions. 
In  the  following  sections,  these  results  will  be  discussed  with  respect  to  the  three  study 
comparisons  made  in  these  analyses: 

• different  variable  sets  (e.g.,  uses  of  job  descriptor  data), 

• different  clustering  procedures,  and 

• different  samples. 

Tables  14-2  through  14-4  present  cluster-solution  convergence  information  which  parallels  the 
above-noted  comparisons. 
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All  of  the  results  will  be  illustrated  with  examples  from  the  analyses.  The  resulting  job 
families  based  on  the  29  incumbent  data  and  the  29  analysts  data  are  shown  at  the  end  of  this 
chapter.  Figures  14-1  through  14-10  present  the  incumbent-based  job  families  across  the  various 
descriptor  usage  and  clustering  procedure  conditions.  Similarly,  Figures  14-1 1 through  14-20 
present  the  analysts-based  job  families.  Figure  14-21  presents  a segment  of  a job  family  solution 
based  on  the  1,122  analyst  data. 


Variable  Set  Comparisons 

The  first  comparison  concerned  various  ways  to  use  the  job  descriptors.  The  set  of 
descriptors  were  used  either  alone  (e.g.,  skills  only)  or  in  combination.  The  different  variations  of 
combining  the  descriptors  included  using  them  all  together  or  in  a two-stage  process  whereby 
clustering  occurred  on  skills  (GW  As)  first  and  then  GW  As  (skills)  second.  As  shown  in  Table 
14-1,  using  the  descriptors  in  a two-stage  approach  provided  better  results  across  the  set  of 
criteria.  In  general,  the  two-stage  approaches  had  higher  percentage  of  correct  reclassification 
rates  and  higher  effect  size  estimates.  The  MSE  estimates  for  the  two-stage  approaches  were  also 
lower  than  when  skills  and  GW  As  were  combined  in  clustering. 

These  results  are  particularly  pronounced  with  the  sample  of  1,122  occupations  rated  by 
occupational  analysts.  Here,  the  reclassification  rates  generally  increased  when  going  from  using 
the  descriptors  in  isolation  to  using  them  in  the  two-stage  approach.  This  illustrates  that  the  two- 
staged  approaches  provide  a better  initial  cluster  solution  that  requires  relatively  less 
readjustment  with  the  confirmatory  k-means  procedure.  The  effect  size  estimates  also  improved 
in  the  two-staged  approach,  indicating  greater  cluster  differentiation.  As  for  the  MSE,  using 
GW  As  only  actually  produced  the  smallest  MSE  estimates  using  the  1,122  analysts  data. 
However,  given  that  using  GW  As  alone  yielded  a lower  effect  size  when  compared  to  the  two- 
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stage  approaches,  it  is  evident  that  the  cluster  solution  based  on  GW  As  did  not  increase  cluster 
differentiation  despite  producing  low  within-cluster  variability.  Finally,  this  pattern  of  results 
also  occurred  with  the  smaller  sample  of  analysts’  and  incumbents’  ratings,  although  it  is  more 
pronounced  in  the  larger  analysts  sample. 

In  making  comparisons  among  different  ways  to  use  job  descriptor  data,  it  is  clear  that 
each  of  the  five  variable  set  approaches  examined  in  this  study  provides  a means  for  producing 
clearly  defined  job  families  based  on  the  statistical  criteria.  Although  the  two-stage  approaches 
yielded  better  results  across  the  statistical  criteria,  the  other  approaches  yielded  excellent  results 
in  an  absolute  sense.  Hence,  if  the  purpose  for  forming  job  families  dictated  a need  to  utilize 
skills  or  GW  As  alone,  the  respective  0*NET  measures  would  still  produce  clearly  differentiated 
job  families.  For  example,  Figure  14-3  shows  five  interpretable  job  clusters  obtained  from 
analyses  using  the  46  basic  and  cross-functional  skills.  The  first  cluster  consists  of  occupations . 
best  classified  as  managerial  in  nature.  Computer  programmers,  which  are  relatively  distinct  from 
the  remaining  occupations,  form  the  second  cluster.  The  third  cluster  consists  of  a variety  of 
public  service  jobs.  The  fourth  cluster  consists  of  occupations  involving  material  handling  and 
transportation.  Finally,  the  fifth  cluster  consists  of  occupations  involving  heavy  equipment  use, 
maintenance,  and  repair. 

Figure  14-9  presents  results  from  the  two-stage  analysis  obtained  by  clustering  GWAs 
within  each  of  the  five  skill-based  clusters  described  above  (see  Figure  14-3).  As  shown  in  Figure 
14-9,  the  four  skill-based  clusters  with  more  than  one  occupation  were  further  partitioned  to  yield 
1 1 final  clusters.  These  final  clusters  represent  a differentiation  of  each  original  skill-based 
cluster  into  groups  relatively  homogenous  with  respect  to  work  activities.  For  example,  the  fourth 
skill  cluster  in  Figure  14-3  combined  occupations  involving  material  handling  and  transportation. 
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This  cluster  was  partitioned  in  Figure  14-9  into  one  cluster  of  occupations  involving  material 
handling  (cluster  8,  Figure  14-9)  and  a second  cluster  of  occupations  involving  transportation 
(cluster  9,  Figure  14-9). 

Clustering  Procedure  Comparisons 

The  second  comparison  concerned  the  specific  agglomerative  clustering  procedure  used 
for  these  data.  As  shown  in  Table  14-1,  there  are  comparisons  of  Ward’s  method  to  the  within- 
groups  average  linkage  method  across  the  other  factors  in  the  study  design.  Each  of  these 
clustering  procedures  share  a similar  heuristic  for  forming  the  clusters.  Namely,  they  seek  to 
ensure  within-cluster  homogeneity,  although  each  approaches  this  objective  differently.  Ward’s 
is  based  on  minimizing  within-cluster  variability,  whereas  within-groups  average  linkage  is  based 
on  minimizing  distance  from  the  cluster  centroid. 

Overall,  both  clustering  procedures  adequately  differentiated  the  occupations.  Looking 
across  Table  14-1,  the  Ward’s  procedure  tended  to  produce  better  defined  job  families,  especially 
for  the  1,122  analysts  data,  in  that  the  MSE  was  generally  smaller  and  the  effect  sizes  larger, 
particularly  with  the  1,122  analysts  data.  Hence,  the  job  families  were  more  homogeneous  within 
clusters,  as  well  as  better  differentiated  between  clusters.  There  were  exceptions,  however,  to  this 
pattern  of  results  across  both  descriptor  set  and  sample.  All  six  exceptions  to  this  pattern 
occurred  with  the  smaller  analysts  and  incumbents  databases. 

A second  caveat  about  the  agglomerative  clustering  procedures  involves  the 

I 

reclassification  rate  calculations.  Cluster  solutions  based  on  Ward’s  method  appear  to  have 
required  fewer  adjustments,  as  indicated  by  their  higher  reclassification  rates.  These  higher 
classification  rates,  however,  may  be  due  to  the  relation  between  Ward’s  procedure  and  the 
MacQueen  (1967)  k-means  procedure.  Specifically,  both  procedures  use  a variance  minimization 
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criteria  in  which  occupations  are  assigned  to  clusters  such  that  the  variability  within  the  cluster  is 
minimized  and  the  variability  between  clusters  is  maximized.  This  correspondence  between  these 
algorithms  creates  the  possibility  that  mathematical  optimization  between  these  methods  partly 
accounts  for  the  better  reclassification  rates  with  Ward’s. 

Despite  these  caveats,  both  Ward’s  and  within-groups  average  linkage  methods  resulted 
in  clear  occupation  differentiation.  Inspection  of  the  figures  shows  that,  within  variable 
descriptor  sets,  the  final  cluster  solutions  were  similar.  For  example.  Figures  14-1 1 and  14-12 
represent  the  cluster  solutions  obtained  from  analysts’  ratings  of  the  42  GW  As  using  Ward’s  and 
within-groups  average  linkage  methods,  respectively.  Twenty-seven  of  the  29  occupations  were 
classified  consistently  across  the  two  methods  (see  Cluster  Solution  Convergence  section). 

Sample  Comparisons 

The  final  study  comparison  made  in  this  analysis  concerned  the  quality  of  the  cluster 
solutions  produced  across  different  samples.  These  comparisons  were  made  primarily  on  the 
smaller  data  sets  because  they  contained  ratings  on  the  same  29  occupations.  Unlike  the  previous 
comparisons,  the  analysts’  and  incumbents’  ratings  produced  markedly  different  results  in  terms 
of  the  statistical  criteria  of  the  resulting  cluster  solutions.  As  shown  in  Table  14-1,  job  families 
using  the  29  common  occupations  evidenced  better  scores  on  the  statistical  criteria  when  based 
on  the  analysts’  ratings.  In  most  instances,  the  MSE  estimates  were  lower,  and  the  effect  size 
estimates  were  higher  among  analysts -based  job  families.  This  pattern  of  results  occurs  whether 
comparisons  are  made  on  the  Ward’s  or  within-groups  average  linkage  based  cluster  solutions. 

Interestingly,  the  reclassification  rates  were  slightly  better  for  clusters  based  on  the 
incumbent  data,  indicating  the  incumbents’  ratings  needed  less  adjustment  after  the  initial 
classification.  However,  this  latter  effect  likely  reflects  the  greater  number  of  clusters  found  when 
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using  the  incumbents’  ratings  (average  number  of  clusters  = 6.7)  than  when  using  the  analysts’ 
ratings  (average  number  of  clusters  = 5.1).  With  only  29  occupations  being  clustered,  extracting 
more  clusters  will  produce  more  single  occupations  clusters  or  clusters  with  only  few  elements. 
This  results  in  less  ambiguity  in  reclassifying  cases  using  the  k-means  procedure. 

Description  of  Complete  Analysts  Data 

The  last  two  columns  of  Table  14-1  show  the  results  from  the  10  cluster  analyses  using 
the  1,122  analysts  data.  As  noted  above,  all  the  descriptors,  singularly  and  combined,  provide 
good  differentiation  of  the  occupations  using  both  Ward’s  and  within-groups  average  linkage 
clustering  methods.  In  fact,  thirty  to  37%  of  the  clusters  from  the  two-stage  procedures  contained 
only  one  occupation.  Figure  14-21  presents  a subset  of  the  final  181  clusters  comprising  the  job 
family  solution  based  on  clustering  the  46  basic  and  cross-functional  skills  first  and  the  42 
GW  As  second.  These  five  clusters  are  typical  of  the  clusters  obtained  in  the  4 two-stage  analyses. 
Accordingly,  the  two  singleton  clusters  which  are  shown  (Artificial  Breeding  Technicians  and 
Engineering  Managers)  seem  sufficiently  unique  with  respect  to  requisite  broad  skills  and  work 
activities.  The  remaining  three  clusters  illustrate  the  unambiguous  differentiation  that  the  two- 
stage  approach  provides  among  occupations.  For  example,  occupations  in  the  natural  sciences 
(e.g..  Biochemist,  Biophysicists)  form  a separate  cluster  from  manage  information  systems 
occupations  (e.g.,  Computer  Engineers,  Systems  Analysts).  Thus,  the  statistical  qualities 
identified  in  the  above  comparison  hold  true  with  the  larger  database  of  occupations. 

Cluster  Solution  Convergence 

Based  on  the  above  comparisons,  it  is  evident  that  the  various  cluster  solutions  have  good 
statistical  properties.  In  this  section,  we  present  some  comparisons  of  these  different  solutions  in 
terms  of  their  similarities  and  differences,  or  convergence.  These  cluster  convergence 
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comparisons  paralleled  those  made  above  in  terms  of  cluster  solution  quality  (e.g.,  across 
variable  sets,  methods,  and  samples).  We  made  the  convergence  assessments  on  the  basis  of  the 
Cramer’s  V statistic,  which  provides  an  index  of  association  between  two  nominal  variables 
(Hays,  1 988).  This  statistic  ranges  from  0 to  1 , with  1 indicating  maximal  association  between 
nominal  variables  and  0 indicating  complete  independence. 

The  first  set  of  cluster  convergence  tests  involved  looking  at  the  extent  to  which  cluster 
solutions  based  on  different  variable  sets  were  associated.  Specifically,  we  made  three  descriptor 
set  comparisons.  First,  we  compared  those  cluster  solutions  based  on  GW  As  only  and  skills  only 
with  each  other  and  the  GW  As  and  skills  combined  solutions.  Second,  we  compared  the  GW  As 
and  skills  combined  solutions  to  both  the  two-stage  solutions.  Note,  however,  that  we  did  not 
compare  GW  As  with  the  GW  As,  then  skills  two-stage  solution  or  the  skills  with  the  skills,  then 
GW  As  two-stage  solution.  These  comparison  resulted  in  inflated  Cramer’s  V estimates  because 
of  the  perfect  association  between  the  GW  As  (skills)  only  solution  and  the  first  stage  of  the 
GW  As  (skills),  then  skills  (GW  As)  two-stage  solution.  Finally,  we  compared  the  two-stage 
solutions  with  each  other  and  the  GW  As  and  skills  combined  solutions. 

As  shown  in  table  14-2,  there  was  moderate  to  strong  association  across  the  cluster 
solutions  based  on  different  variable  sets.  The  weakest  level  of  convergence  occurred  when 
comparing  the  cluster  solutions  based  on  GW  As  only  and  skills  only  using  the  29  incumbent  data 
and  the  1 ,122  analysts  data.  This  degree  of  divergence  is  expected  given  the  differences  between 
job  attributes  (i.e.,  GW  As)  and  worker  attributes  (i.e.,  skills)  on  which  clustering  occurred.  For 
example,  the  Cramer’s  V estimate  for  the  GW  As  versus  skills  variable  set  comparison  using  the 
29  incumbent  data  was  only  .59  (Table  14-2).  Inspection  of  the  corresponding  cluster  solutions 
(Figures  14-1  and  14-3)  pinpoint  the  nature  of  these  differences. 
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In  the  GW  As-based  cluster  solution  (Figure  14-1),  Computer  Programmers  occupied  a 
cluster  with  occupations  performing  similar  work  functions,  such  as  Bookkeepers,  Police  Patrol 
Officers,  and  Office  Clerks.  Some  of  the  shared  GW  As  among  these  occupations  might  include 
processing  information,  analyzing  data  or  information,  and  interacting  with  computers.  However, 
Computer  Programmers  require  a distinct  set  of  skills  (e.g.,  programming)  from  those  required  of 
Bookkeepers,  Patrol  Officers,  and/or  Office  Clerks.  The  skill-based  cluster  solution  (Figure  14-3) 
shows  that  computer  programmers  were  indeed  placed  into  a unique  cluster. 

A final  note  of  interests  concerns  the  relationship  between  the  two  cluster  solutions  based 
on  the  two-stage  approaches.  As  shown  in  Table  14-2,  these  solutions  show  good  convergence 
using  solutions  based  on  the  smaller  databases,  but  are  less  associated  using  the  larger  analysts 
database.  In  the  latter  instance,  the  Cramer’s  V estimate  is  only  .64  for  both  Ward’s  and  within- 
groups  average  linkage  solutions.  Thus,  although  the  two-stage  solutions  provided  statistically.  , 
sound  clusters,  these  clusters  vary  depending  on  whether  skills  or  GW  As  are  cluster  first.  This 
finding  simply  parallels  the  lower  convergence  between  the  GW  As  only  and  skills  only  based 
solutions. 

The  second  set  of  cluster  convergence  tests  involved  assessing  the  extent  to  which  cluster 
solutions  based  on  the  Ward’s  and  within-groups  average  linkage  methods  converged.  Table  14-3 
presents  the  results  from  these  comparisons.  As  shown,  the  cluster  solutions  did  not  vary  greatly 
as  a function  of  using  either  method.  All  Cramer’s  V estimates  equal  or  exceed  the  .70  level, 
indicating  above  average  association  among  the  various  solution  comparisons.  The  convergence 
among  cluster  solutions  tends  to  be  larger  in  the  smaller  samples  versus  the  larger  analysts 
sample.  One  possible  explanation  for  this  trend  is  that  a greater  number  of  clusters  are  being 
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generated  among  a more  diverse  set  of  occupations  in  the  larger  database.  This  creates  the 
opportunity  for  more  variability  in  the  clustering  procedures  and  final  cluster  solutions. 

The  final  set  of  cluster  convergence  analyses  compared  the  cluster  solutions  based  on 
incumbents’  ratings  with  those  based  on  analysts’  ratings,  using  the  29  occupation  data  only.  As 
shown  in  Table  14-4,  the  incumbents  and  analysts  based  solutions  tend  to  converge  more  when 
using  both  GW  As  and  skills,  regardless  of  whether  used  together  or  in  a two-staged  approach. 
The  greatest  level  of  convergence  occurs  when  using  the  GWAs  and  skills  together.  Inspection  of 
Figures  14-5  and  14-15  illustrate  the  nature  of  this  convergence.  Using  the  Ward’s  method,  the 
analyst  data  resulted  in  three  clusters  and  the  incumbents  data  resulted  in  6 clusters.  However, 
close  inspection  of  these  tables  reveals  that  the  first  3 clusters  in  the  incumbent  solution 
correspond  to  the  first  cluster  in  the  analysts  solution,  minus  Sales  Representatives  and. 

Packaging  and  Filling  Machine  Operators.  Similarly,  the  last  cluster  from  each  solution  shares  all 
but  one  element.  And  by  extension,  incumbent  clusters  4 and  5 (Figure  14-5)  largely  overlap  with 
analysts  cluster  2 (Figure  14-15). 

Across  all  the  convergence  comparisons  in  Tables  14-2,  14-3,  and  14-4,  the  overall  level 
of  convergence  in  clustering  solutions  was  quite  high.  This  finding  bodes  well  for  the  assessment 
of  the  stability  of  clustering  solutions  based  on  combinations  of  basic  and  cross-functional  skills 
and  GWAs. 

Conclusion 

The  results  from  this  study  paint  a rather  straight-forward  and  enthusiastic  picture:  The 
basic  and  cross-functional  skills  and  GWAs  measures  from  the  0*NET  content  model  provide 
excellent  job  description  data.  This  finding  is  robust  in  that  job  families  based  on  these  job 
descriptors  where  both  statistically  sound  and  easily  interpretable.  The  overall  high  level  of 
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coherence  and  job  differentiation  in  the  job  families  did  not  vary  substantially  across  various 
parameters  of  clustering  known  to  affect  results  of  job  family  creation.  Furthermore,  there  is  a 
nigh  degree  of  association  between  the  resulting  cluster  solutions,  particularly  those  based  on 
both  GWAs  and  skills.  Based  on  these  findings,  the  0*NET  job  descriptors  appear  to  be 
accomplishing  their  objective  of  allowing  cross  occupation  comparisons. 


1279 


14-28  Norris,  Baughman,  Cooke,  Peterson,  & Mumford 

References 

Colihan,  J.,  & Burger,  G.  K.  (1995).  Constructing  job  families:  An  analysis  of 
quantitative  techniques  used  for  grouping  jobs.  Personnel  Psychology.  48.  563-586. 

Cornelius,  E.  T.,  Carron,  T.  J.,  & Collins,  M.  N.  (1979).  Job  analysis  models  and  job 
classification.  Personnel  Psychology.  32,  693-708. 

Cronbach,  L.  J.,  & Gleser,  G.  C.  (1953).  Assessing  similarity  between  profiles. 
Psychological  Bulletin  50  456-473. 

Everitt,  B.  (1993).  Cluster  analysis  (3rd  ed.).  New  York:  Halsted. 

Gorsuch,  R.  L.  (1983).  Factor  Analysis  (2nd  ed.).  Hillsdale,  NJ:  Lawrence  Erlbaum 
Associates,  Publishers. 

Harvey,  R.  J.  (1986).  Quantitative  approaches  to  job  classification:  A review  and  critique. 
Personnel  Psychology.  39  267-289. 

Harvey,  R.  J.  (1990).  Job  analysis.  In  M.  D.  Dunnette  & L.  M.  Hough  (Eds.),  Handbook 
of  industrial  and  organizational  psychology:  Vol.  1 (2nd  ed.,  pp.  70-163).  Palo  Alto,  CA: 
Consulting  Psychologists  Press,  Inc. 

Hay,  W.  L.  (1988).  Statistics  (4th  ed.).  Fort  Worth,  TX:  Holt,  Rinehart  and  Winston,  Inc. 

Kaufmann,  L.,  & Rousseeuw,  P.  J.  (1990).  Finding  groups  in  data:  An  intrndnrtinn  tn 
cluster  analysis.  New  York:  Wiley. 

MacQueen,  J.  (1967).  Some  methods  for  classification  and  analysis  of  multivariate 
observations.  In  L.  Le  Cam  & J.  Neyman  (Eds.),  Fifth  Berkeley  Symposium  nf  Mathematics 
Statistics,  and  Probability  (Vol.  1,  pp.  281-297). 

Norusis/SPSS,  Inc.  (1993).  SPSS  for  Windows:  Professional  Statistics  (Release  6.0). 
Chicago:  SPSS,  Inc. 


1230 


Occupation  Classification  14-29 


Pearlman,  K.  (1980).  Job  families:  A review  and  discussion  of  their  implications  for 
personnel  selection.  Psychological  Bulletin.  87. 80-107. 

Peterson,  N.,  Mumford,  M.  D.,  Borman,  W.,  Jeanneret,  R.,  & Fleishman,  E.  A.  (1995). 
Development  of  a Prototype  Occupational  Information  Network  (Q*NET)  Content  Model  (Vols. 
1-2).  Salt  Lake  City,  UT:  Utah  Department  of  Employment  Security. 

Ward,  J.  J.,  & Hook,  M.  E.  (1963).  Application  of  a hierarchical  grouping  procedure  to 
the  problem ‘of  grouping  profiles.  Educational  and  Psychological  Measurement,  23.  69-81. 


Figure  14-1 

Twentv-Nine  Occupations  Clustered  on  Incumbents’  Ratings  of  42  Generalized  Work  Activitites  (Ward’s 
Method! 


Occupation 

Code 

Occupation  Title 

Mean 
Across  42 
GW  As 

Cluster 

! 15005 

Education  Administrators 

4.19 

i 

61005 

Police  and  Detective  Supervisors 

3.87 

1 

19005 

General  Managers  and  Top  Executives 

3.48 

1 

i 32502 

Registered  Nurses 

3.16 

i 

j 51002 

First-Line  Supervisors  and  Manager/Supervisors,  Clerical  and 
Adminstrative  Support  Workers 

3.11 

1 

1 31305 

Teachers,  Elementary  School 

3.09 

1 

49008 

Sales  Representatives,  except  Scientific  and  Related  Products  or 
Services  and  Retail 

3.01 

2 

25105 

Computer  Programmers 

2.99 

2 

63014 

Police  Patrol  Officers 

2.90 

2 

53905 

Teacher  Aides  and  Educational  Assistants,  Clerical 

2.38 

2 

49011 

Salespersons,  Retail 

2.26 

2 

55338 

Bookkeeping,  Accounting,  and  Auditing  Clerks 

2.20 

2 

55347 

General  Office  Clerks 

2.16 

2 

55108 

Secretaries,  except  Legal  and  Medical 

2.10 

2 

53102 

Tellers 

1.87 

2 

55305 

Receptionists  and  Information  Clerks 

1.46 

2 

32902 

Medical  and  Clinical  Laboratory  Technologists 

3.13 

3 

85132 

Maintenance  Repairers,  General  Utility 

2.85 

3 

92974 

Packaging  and  Filling  Machine  Operators  and  Tenders 

2.53 

3 

87902 

Earth  Drillers,  except  Oil  and  Gas 

2.51 

3 

97111 

Bus  Drivers,  School 

2.25 

3 

65026 

Cooks,  Restaurant 

1.72 

3 

97102 

Truck  Drivers,  Heavy  or  Tractor-Trailer 

1.32 

3 

65008 

Waiters  and  Waitresses 

2.34 

4 

49021 

Counter  and  Rental  Clerks 

2.17 

4 

65038 

Food  Preparation  Workers 

1.94 

4 

67005 

Janitors  and  Cleaners,  except  Maids  and  Housekeeping  Cleaners 

1.83 

4 

66008 

Nursing  Aides,  Orderlies,  and  Attendants 

1.57 

4 

49023 

Cashiers 

1.42 

4 

O 
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Figure  14-2 

Twenty-Nine  Occupations  Clustered  First  on  Incumbents*  Ratings  of  42  Generalized  Work  Activitites 
(Within-Groups  Average  Linkages 


Occupation 

Code 

Occupation  Title 

Mean 

Across 

42 

GWAs 

Cluster 

15005 

Education  Administrators 

4.19 

1 

61005 

Police  and  Detective  Supervisors 

3.87 

1 

19005 

General  Managers  and  Top  Executives 

3.48 

1 

32502 

Registered  Nurses 

3.16 

1 

32902 

Medical  and  Clinical  Laboratory  Technologists 

3.13 

1 

51002 

First-Line  Supervisors  and  Manager/Supervisors,  Clerical  and 
Adminstrative  Support  Workers 

3.11 

1 

31305 

Teachers,  Elementary  School 

3.09 

1 

49008 

Sales  Representatives,  except  Scientific  and  Related  Products  or 
Services  and  Retail 

3.01 

1 

25105 

Computer  Programmers 

2.99 

1 

63014 

Police  Patrol  Officers 

2.90 

1 

53905 

Teacher  Aides  and  Educational  Assistants,  Clerical 

2.38 

1 

55338 

Bookkeeping,  Accounting,  and  Auditing  Clerks 

2.20 

1 

55347 

General  Office  Clerks 

2.16 

1 

55108 

Secretaries,  except  Legal  and  Medical  . 

2.10 

1 

53102 

Tellers 

. 1.87 

1 

55305 

Receptionists  and  Information  Clerks 

1.46 

1 

92974 

Packaging  and  Filling  Machine  Operators  and  Tenders 

2.53 

2 

65008 

Waiters  and  Waitresses 

2.34 

2 

49011 

Salespersons,  Retail 

2.26 

2 

49021 

Counter  and  Rental  Clerks 

2.17 

2 

65038 

Food  Preparation  Workers 

1.94 

2 

67005 

Janitors  and  Cleaners,  except  Maids  and  Housekeeping  Cleaners 

1.83 

2 

65026 

Cooks,  Restaurant 

1.72  i 

2 

66008 

Nursing  Aides,  Orderlies,  and  Attendants 

1.57 

2 

49023 

Cashiers 

1.42 

2 

85132 

Maintenance  Repairers,  General  Utility 

2.85 

3 

87902 

Earth  Drillers,  except  Oil  and  Gas 

2.51 

3 

97111 

Bus  Drivers,  School 

2.25 

3 

97102 

Truck  Drivers,  Heavy  or  Tractor-Trailer 

1.32 

3 

erIc 


Figure  14-3 

Twenty-Nine  Occupations  Clustered  on  Incumbents’  Ratings  of  46  Basic  and  Cross-Functional  Skills 
fWithin-Groups  Average  Linkage) 


Occupation 

Code 

Occupation  Title 

Mean 
Across 
46  Skills 

Cluster 

15005 

Education  Administrators 

4.24 

i 

19005 

General  Managers  and  Top  Executives 

4.11 

i 

61005 

Police  and  Detective  Supervisors 

3.74 

i 

49008 

Sales  Representatives,  except  Scientific  and  Related  Products  or  Services 
and  Retail 

. 3.47 

i 

51002 

First-Line  Supervisors  and  Manager/Supervisors,  Clerical  and 
Adminstrative  Support  Workers 

3.42 

i 

25105 

Computer  Programmers 

4.01 

2 

32502 

Registered  Nurses 

3.54 

3 

31305 

Teachers,  Elementary  School 

3.52 

3 

63014 

Police  Patrol  Officers 

2.87 

3 

49011 

Salespersons,  Retail 

2.65 

3 

53905 

Teacher  Aides  and  Educational  Assistants,  Clerical 

2.60 

3 

55108 

Secretaries,  except  Legal  and  Medical 

2.38 

3 

55338 

Bookkeeping,  Accounting,  and  Auditing  Clerks 

2.29 

3 

49021 

Counter  and  Rental  Clerks 

2.02 

3 

55347 

General  Office  Clerks 

1.93 

3 

66008 

Nursing  Aides,  Orderlies,  and  Attendants 

1.91 

3 

49023 

Cashiers 

1.76 

3 

55305 

Receptionists  and  Information  Clerks 

1.69 

3 

53102 

Tellers 

1.49 

3 

65026 

Cooks,  Restaurant 

1.34 

3 

65008 

Waiters  and  Waitresses 

1.31 

3 

32902 

Medical  and  Clinical  Laboratory  Technologists 

4.08 

4 

65038 

Food  Preparation  Workers 

2.23 

4 

97111 

Bus  Drivers,  School 

2.22 

4 

97102 

Truck  Drivers,  Heavy  or  Tractor-Trailer 

2.15 

4 

92974 

Packaging  and  Filling  Machine  Operators  and  Tenders 

2.11 

4 

87902 

Earth  Drillers,  except  Oil  and  Gas 

4.52 

5 

85132 

Maintenance  Repairers,  General  Utility 

3.42' 

5 

67005 

Janitors  and  Cleaners,  except  Maids  and  Housekeeping  Cleaners 

2.15 

5 
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Figure  14-4  • 

Twenty-Nine  Occupations  Clustered  on  Incumbents’  Ratings  of  46  Basic  and  Cross-Functional  Skills 
(Within-Groups  Average  Linkage! 


Occupation 

Code 

Occupation  Tide 

Mean 
Across 
46  Skills 

Cluster 

15005 

Education  Administrators 

4.24 

1 

19005 

General  Managers  and  Top  Executives 

4.11 

1 

61005 

Police  and  Detective  Supervisors 

3.74 

1 

32502 

Registered  Nurses 

3.54 

1 

31305 

Teachers,  Elementary  School 

3.52 

1 

49008 

Sales  Representatives,  except  Scientific  and  Related  Products  or 
Services  and  Retail 

3.47 

1 

51002 

First-Line  Supervisors  and  Manager/Supervisors,  Clerical  and 
Adminstrative  Support  Workers 

3.42 

1 

63014 

Police  Patrol  Officers 

2.87 

1 

49011 

Salespersons,  Retail 

2.65 

1 

53905 

Teacher  Aides  and  Educational  Assistants,  Clerical 

2.60 

1 

55108 

Secretaries,  except  Legal  and  Medical 

2.38 

1 

55338 

Bookkeeping,  Accounting,  and  Auditing  Clerks 

2.29 

1 

97111 

Bus  Drivers,  School 

2.22 

1 

49021 

Counter  and  Rental  Clerks 

2.02 

1 

55347 

General  Office  Clerks 

1.93  | 

1 

66008 

Nursing  Aides,  Orderlies,  and  Attendants 

1.91 

1 

49023 

Cashiers 

1.76 

1 

55305 

Receptionists  and  Information  Clerks 

1.69 

1 

53102 

Tellers 

1.49 

1 

65026 

Cooks,  Restaurant 

1.34 

1 

65008 

Waiters  and  Waitresses 

1.31 

1 

25105 

Computer  Programmers 

4.01 

2 

32902 

Medical  and  Clinical  Laboratory  Technologists  j 

4.08 

3 

67005 

Janitors  and  Cleaners,  except  Maids  and  Housekeeping  Cleaners 

2.15 

4 

87902 

Earth  Drillers,  except  Oil  and  Gas 

4.52 

5 

85132 

Maintenance  Repairers,  General  Utility  | 

3.42 

5 

65038 

Food  Preparation  Workers 

2.23 

6 

97102 

Truck  Drivers,  Heavy  or  Tractor-Trailer 

2.15 

6 

92974 

Packaging  and  Filling  Machine  Operators  and  Tenders 

2.11 

6 
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Figure  14-5 

Twenty-Nine  Occupations  Clustered  on  Incumbents*  Ratings  of  46  Basic  and  Cross-Functional  Skills  and 
42  Generalized  Work  Activates  (Ward's  Method) 


Occupation 

Code 

Occupation  Title 

Mean 
Across  42 
GW  As  and 
46  Skills 

Cluster 

15005 

Education  Administrators 

4.21 

i 

61005 

Police  and  Detective  Supervisors 

3.81 

i 

19005  1 

General  Managers  and  Top  Executives 

3.80 

i 

32502 

Registered  Nurses 

3.35 

i 

31305 

Teachers,  Elementary  School 

3.30 

i 

51002 

First-Line  Supervisors  and  Manager/Supervisors,  Clerical  and 
Adminstrative  Support  Workers 

3.27 

i 

49008 

Sales  Representatives,  except  Scientific  and  Related  Products  or 
Services  and  Retail 

3.24 

i 

25105 

Computer  Programmers 

3.50 

2 

32902 

Medical  and  Clinical  Laboratory  Technologists 

3.60 

3 

92974 

Packaging  and  Filling  Machine  Operators  and  Tenders 

2.32 

3 

65026 

Cooks,  Restaurant 

1.53 

3 

63014 

Police  Patrol  Officers 

2.88 

4 

49011 

Salespersons,  Retail 

2.45 

4 

49021 

Counter  and  Rental  Clerks 

2.10 

4 

65008 

Waiters  and  Waitresses 

1.83 

4 

66008 

Nursing  Aides,  Orderlies,  and  Attendants 

1.74 

4 

53102 

Tellers 

1.68 

4 

49023 

Cashiers 

1.59 

4 

53905  1 

Teacher  Aides  and  Educational  Assistants,  Clerical 

2.49 

5 

55338 

Bookkeeping,  Accounting,  and  Auditing  Clerks 

2.24 

5 

55108 

Secretaries,  except  Legal  and  Medical 

2.24 

5 

55347 

General  Office  Clerks 

2.04 

5 

55305 

Receptionists  and  Information  Clerks 

1.58 

5 

87902 

Earth  Drillers,  except  Oil  and  Gas 

3.51 

6 

85132 

Maintenance  Repairers,  General  Utility 

3.13 

6 

97111 

Bus  Drivers,  School 

2.23 

6 

65038 

Food  Preparation  Workers 

2.08 

6 

67005 

Janitors  and  Cleaners,  except  Maids  and  Housekeeping  Cleaners 

1.99 

6 

97102 

Truck  Drivers,  Heavy  or  Tractor-Trailer 

1.73 

6 
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Figure  14-6 

Twenty-Nine  Occupations  Clustered  on  Incumbents’  Ratings  of  46  Basic  and  Crnss-Fnnr.fi onal  Skills  and 
42  Generalized  Work  Activitites  (Within-Groups  Average  Linkagei 


Occupation 

Code 

Occupation  Title 

Mean 
Across  42 
GW  As 
and  46 
Skills 

Cluster 

15005 

Education  Administrators 

4.21 

1 

61005 

Police  and  Detective  Supervisors 

3.81 

1 

19005 

General  Managers  and  Top  Executives 

3.80 

1 

32502 

Registered  Nurses 

3.35 

1 

31305 

Teachers,  Elementary  School 

1 3.30 

1 

51002 

First-Line  Supervisors  and  Manager/Supervisors,  Clerical  and 
Adminstrative  Support  Workers 

3.27 

1 

49008 

Sales  Representatives,  except  Scientific  and  Related  Products  or 
Services  and  Retail 

3.24 

1 

25105 

Computer  Programmers 

3.50 

2 

32902 

Medical  and  Clinical  Laboratory  Technologists 

3.60 

3 

63014 

Police  Patrol  Officers 

2.88 

4 

53905 

Teacher  Aides  and  Educational  Assistants,  Clerical 

2.49 

4 

49011 

Salespersons,  Retail 

2.45 

4 

92974 

Packaging  and  Filling  Machine  Operators  and  Tenders 

2.32 

4 

55338 

Bookkeeping,  Accounting,  and  Auditing  Clerks 

2.24 

4 

55108 

Secretaries,  except  Legal  and  Medical 

2.24 

4 

97111 

Bus  Drivers,  School 

2.23 

4 

49021 

Counter  and  Rental  Clerks 

2.10 

4 

65038 

Food  Preparation  Workers 

2.08 

4 

55347 

General  Office  Clerks 

2.04 

4 

65008 

Waiters  and  Waitresses 

1.83 

4 

66008 

Nursing  Aides,  Orderlies,  and  Attendants 

1.74 

4 

53102 

Tellers 

1.68 

4 

49023 

Cashiers 

1.59 

4 

55305 

Receptionists  and  Information  Clerks  _ 

1.58 

4 

67005 

Janitors  and  Cleaners,  except  Maids  and  Housekeeping  Cleaners 

1.99 

• 5 

65026 

Cooks,  Restaurant 

1.53 

6 

85132 

Maintenance  Repairers,  General  Utility 

3.13 

7 

87902 

Earth  Drillers,  except  Oil  and  Gas 

3.51 

8 

97102 

Truck  Drivers,  Heavy  or  Tractor-Trailer 

1.73 

9 1 

O 
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Figure  14-7 

Twenty-Nine  Occupations  Clustered  First  on  Incumbents'  Ratings  of  42  Generalized  Work  Activitites; 
Second  on  46  Basic  and  Cross-Functional  Skills  (Ward’s  Method) ; 


Occupation 

Code 

Occupation  Title 

Mean 
Across 
46  Skills 

Cluster 

15005 

Education  Administrators 

4.24 

1 

19005 

General  Managers  and  Top  Executives 

4.11 

1 

61005 

Police  and  Detective  Supervisors 

3.74 

i 

51002 

First-Line  Supervisors  and  Manager/Supervisors,  Clerical  and 
Adminstrative  Support  Workers 

3.42 

i 

32502 

Registered  Nurses 

3.54 

2 

31305 

Teachers,  Elementary  School 

3.52 

2 

49008 

Sales  Representatives,  except  Scientific  and  Related  Products  or  Services 
and  Retail 

3.47 

3 

63014 

Police  Patrol  Officers 

2.87 

3 

49011 

Salespersons,  Retail 

2.65 

3 

53905 

Teacher  Aides  and  Educational  Assistants,  Clerical 

2.60 

3 

55108 

Secretaries,  except  Legal  and  Medical 

2.38 

3 

55338 

Bookkeeping,  Accounting,  and  Auditing  Clerks 

2.29 

3 

55347 

General  Office  Clerks 

1.93 

3 

55305 

Receptionists  and  Information  Clerks 

1.69 

3 

53102 

Tellers 

1.49 

3 

25105 

Computer  Programmers 

4.01 

4 

32902 

Medical  and  Clinical  Laboratory  Technologists 

4.08 

5 

92974 

Packaging  and  Filling  Machine  Operators  and  Tenders 

2.11 

5 

65026 

Cooks,  Restaurant 

1.34 

' 6 

97111 

Bus  Drivers,  School 

2.22 

7 

97102 

Truck  Drivers,  Heavy  or  Tractor-Trailer 

2.15 

7 

87902 

Earth  Drillers,  except  Oil  and  Gas 

4.52 

8 

85132 

Maintenance  Repairers,  General  Utility 

3.42 

8 

49021 

Counter  and  Rental  Clerks 

2.02 

9 

66008 

Nursing  Aides,  Orderlies,  and  Attendants 

1.91 

9 

49023 

Cashiers 

1.76 

9 

65008 

Waiters  and  Waitresses 

1.31 

9 

65038 

Food  Preparation  Workers 

2.23 

10 

67005 

Janitors  and  Cleaners,  except  Maids  and  Housekeeping  Cleaners 

2.15 

11 

1288 


Figure  14-8 


Twenty-Nine  Occupations  Clustered  First  on  Incumbents'  Ratings  of  42  Generalized  Work  Activities: 
Second  on  46  Basic  and  Cross-Functional  Skills  fWithin-Groups  Average  Linkage) 


Occupation 

Code 

Occupation  Title 

Mean 
Across 
46  Skills 

Cluster 

15005 

Education  Administrators 

4.24 

1 

19005 

General  Managers  and  Top  Executives 

4.11 

1 

61005 

Police  and  Detective  Supervisors 

3.74 

1 

31305 

Teachers,  Elementary  School 

3.52 

1 

49008  • 

Sales  Representatives,  except  Scientific  and  Related  Products  or  Services 
and  Retail 

3.47 

1 

51002 

First-Line  Supervisors  and  Manager/Supervisors,  Clerical  and 
Adminstrative  Support  Workers 

3.42 

1 

32502 

Registered  Nurses 

3.54 

2 

63014 

Police  Patrol  Officers 

2.87 

2 

53905 

Teacher  Aides  and  Educational  Assistants,  Clerical 

2.60 

2 

55108 

Secretaries,  except  Legal  and  Medical 

2.38 

2 

55338 

Bookkeeping,  Accounting,  and  Auditing  Clerks 

2.29 

2 

55347 

General  Office  Clerks 

1.93 

2 

55305 

Receptionists  and  Information  Clerks 

1.69 

2 

53102 

Tellers 

1.49 

2 

25105 

Computer  Programmers 

4.01 

3 

32902 

Medical  and  Clinical  Laboratory  Technologists 

4.08 

4 

92974 

Packaging  and  Filling  Machine  Operators  and  Tenders 

2.11 

4 

49011 

Salespersons,  Retail 

2.65 

5 

49021 

Counter  and  Rental  Clerks 

2.02 

5 

66008 

Nursing  Aides,  Orderlies,  and  Attendants 

1.91 

5 

49023 

Cashiers 

1.76 

5 

1 65026 

Cooks,  Restaurant 

1.34 

5 

65008 

Waiters  and  Waitresses 

1.31 

5 

65038 

Food  Preparation  Workers 

2.23 

6 

67005 

Janitors  and  Cleaners,  except  Maids  and  Housekeeping  Cleaners 

2.15 

7 

' 

97111 

Bus  Drivers,  School 

2.22 

8 

97102 

Truck  Drivers,  Heavy  or  Tractor-Trailer 

2.15 

8 

87902 

Earth  Drillers,  except  Oil  and  Gas 

4.52 

9 

85132 

Maintenance  Repairers,  General  Utility 

3.42 

9 

Figure  14-9 

Twenty-Nine  Occupations  Clustered  First  on  Incumbents*  Ratings  of  46  Basic  and  Cross-Functional  Skills; 
Second  on  42  Generalized  Work  Activitites  (Ward’s  Method') 


Occupation 

Code 

Occupation  Title 

i 

Mean 

Across 

42 

GWAs 

Cluster 

15005 

Education  Administrators 

4.19 

1 

19005 

General  Managers  and  Top  Executives 

3.48 

1 

51002 

First-Line  Supervisors  and  Manager/Supervisors,  Clerical  and 
Adminstrative  Support  Workers 

3.11 

1 

61005 

Police  and  Detective  Supervisors 

3.87 

2 

49008 

Sales  Representatives,  except  Scientific  and  Related  Products  or  Services 
and  Retail 

3.01 

3 

25105 

Computer  Programmers 

2.99 

4 

32502 

Registered  Nurses 

3.16 

5 

31305 

Teachers,  Elementary  School 

3.09 

5 

53905 

Teacher  Aides  and  Educational  Assistants,  Clerical 

2.38 

5 

55338 

Bookkeeping,  Accounting,  and  Auditing  Clerks 

2.20 

5 

55347 

General  Office  Clerks 

2.16  1 

5 

55108 

Secretaries,  except  Legal  and  Medical 

2.10 

‘ 5 

65026 

Cooks,  Restaurant 

1.72 

6 

63014 

Police  Patrol  Officers 

2.90 

7 

65008 

Waiters  and  Waitresses 

2.34 

7 

49011 

Salespersons,  Retail 

2.26 

7 

49021 

Counter  and  Rental  Clerks 

2.17 

7 

53102 

Tellers 

1.87 

7 

66008 

Nursing  Aides,  Orderlies,  and  Attendants 

1.57 

7 

55305 

Receptionists  and  Information  Clerks 

1.46 

7 

49023 

Cashiers 

1.42 

7 

32902 

Medical  and  Clinical  Laboratory  Technologists 

3.13 

8 

92974 

Packaging  and  Filling  Machine  Operators  and  Tenders 

2.53 

8 

65038 

Food  Preparation  Workers 

1.94 

8 

97111 

Bus  Drivers,  School 

2.25 

9 

97102 

Truck  Drivers,  Heavy  or  Tractor-Trailer 

1.32 

9 

67005 

Janitors  and  Cleaners,  except  Maids  and  Housekeeping  Cleaners 

1.83 

10 

85132 

Maintenance  Repairers,  General  Utility 

2.85 

11 

87902 

Earth  Drillers,  except  Oil  and  Gas 

I 2.51 

11 

1230 


Figure  14-10 

Twenty-Nine  Occupations  Clustered  First  on  Incumbents’  Ratines  of  46  Basic  and  Cross-Functional  Skills: 
Second  on  42  Generalized  Work  Activitites  fWithin-Grouns  Average  Linkage') 


Occupation 

Code 

Occupation  Title 

Mean 

Across 

42 

GWAs 

' Cluster 

15005 

Education  Administrators 

4.19 

i 

61005 

Police  and  Detective  Supervisors 

3.87 

i 

19005 

General  Managers  and  Top  Executives 

3.48 

i 

32502 

Registered  Nurses 

3.16 

i 

51002 

First-Line  Supervisors  and  Manager/Supervisors,  Clerical  and 
Adminstrative  Support  Workers 

3.11 

i 

31305 

Teachers,  Elementary  School 

3.09 

i 

49008 

Sales  Representatives,  except  Scientific  and  Related  Products  or  Services 
and  Retail 

3.01 

i 

53905 

Teacher  Aides  and  Educational  Assistants,  Clerical 

2.38 

1 2 

55338 

Bookkeeping,  Accounting,  and  Auditing  Clerks 

2.20 

! 2 

55347 

General  Office  Clerks 

2.16 

2 

55108 

Secretaries,  except  Legal  and  Medical 

2.10 

2 

55305 

Receptionists  and  Information  Clerks 

1.46 

2 

63014 

Police  Patrol  Officers 

2.90 

3 

65008 

Waiters  and  Waitresses 

2.34 

3 

49011 

Salespersons,  Retail 

2.26 

3 

49021 

Counter  and  Rental  Clerks 

2.17 

3 

53102 

^Tellers 

1.87 

3 

49023 

Cashiers 

1.42 

3 

65026 

Cooks,  Restaurant 

1.72 

4 

66008 

Nursing  Aides,  Orderlies,  and  Attendants 

1.57 

5 

97111 

Bus  Drivers,  School 

2.25 

6 

25105 

Computer  Programmers 

2.99  | 

7 

32902 

Medical  and  Clinical  Laboratory  Technologists 

3.13 

8 

67005 

Janitors  and  Cleaners,  except  Maids  and  Housekeeping  Cleaners 

1.83  | 

9 

85132 

Maintenance  Repairers,  General  Utility 

2.85 

10 

87902 

Earth  Drillers,  except  Oil  and  Gas 

2.51 

10 

92974 

Packaging  and  Filling  Machine  Operators  and  Tenders 

2.53 

11 

65038 

Food  Preparation  Workers 

1.94 

11 

97102 

Truck  Drivers,  Heavy  or  Tractor-Trailer 

1.32 

12 

1291 


Figure  14-11 

Twenty-Nine  Occupations  Clustered  First  on  Analysts'  Ratings  of  42  Generalized  Work  Activitites 


(Ward’s  Metl 

lod) 

Occupation 

Code 

Occupation  Title 

Mean 
Across  42 
GWAs 

Cluster 

19005 

General  Managers  and  Top  Executives 

4.16 

1 

15005 

Education  Administrators 

3.88 

1 

61005 

Police  and  Detective  Supervisors 

3.77 

1 

32502 

Registered  Nurses 

3.33 

1 

25105 

Computer  Programmers 

3.30 

1 

32902 

Medical  and  Clinical  Laboratory  Technologists 

3.19 

1 

31305 

Teachers,  Elementary  School 

3.13 

1 

51002 

First  Line  Supervisors  and  Managers,  Clerical/Admin. 

2.90 

1 

65026 

Cooks,  Restaurant 

2.44 

1 

53905 

Teacher  Aides  and  Educational  Assistants,  Clerical 

2.22 

1 

55338 

Bookkeeping,  Accounting,  and  Auditing  Clerks 

1.93 

1 

63014 

Law  Enforcement  Officers 

3.05 

2 

49008 

Sales  Representatives,  except  Scientific  and  Related  Products  or 
Services  and  Retail 

2.62 

2 

53102 

Tellers 

1.92 

2 

49011 

Salespersons,  Retail 

1.85 

2 

55108 

Secretaries,  except  Legal  and  Medical 

1.82 

2 

49021 

Stock  Clerks,  Sales  Floor 

1.81 

2 

65008 

Waiter /Waitress 

1.75 

2 

55305 

Receptionists  and  Information  Clerks 

1.74 

2 

55347 

General  Office  Clerks 

1.74 

2 

66008 

Nursing  Aides,  Orderlies,  and  Attendants 

1.72 

2 

49023 

Cashiers 

1.67 

2 

97111 

Bus  Drivers,  School 

1.32 

t 2 

85132 

Maintenance  Repairers,  General  Utility 

2.09 

3 

97102 

Truck  Drivers,  Heavy  and  Tractor-Trailer 

1.84 

3 

87902 

Earth  Drillers,  except  Oil  or  Gas 

1.70 

3 

67005 

Janitors  and  Cleaners,  except  Maids  and  Housekeeping  Cleaners 

1.52 

3 

65038 

Food  Preparation  Workers 

1.24 

3 

92974 

Packaging  and  Filling  Machine  Operators  and  Tenders 

1.23 

3 

Figure  14-12 

TwentY^Nine  Occupations  Clustered  on  Analysts’  Ratings  of  42  Generalized  Work  Activities  (Within- 
Groups  Average  Linkages 


Occupation 

Code 

Occupation  Title 

Mean 
Across  42 
GWAs 

Cluster 

19005 

General  Managers  and  Top  Executives 

4.16  . 

1 

15005 

Education  Administrators 

3.88 

1 

61005 

Police  and  Detective  Supervisors 

3.77 

1 

32502 

Registered  Nurses 

3.33 

1 

25105 

Computer  Programmers 

3.30 

1 

32902 

Medical  and  Clinical  Laboratory  Technologists 

3.19 

1 

31305 

Teachers,  Elementary  School 

3.13 

1 

51002 

First  Line  Supervisors  and  Managers,  Clerical/Admin. 

2.90 

1 

65026 

Cooks,  Restaurant 

2.44 

1 

63014 

Law  Enforcement  Officers 

3.05 

2 

49008 

Sales  Representatives,  except  Scientific  and  Related  Products  or 
Services  and  Retail 

2.62 

2 

53905 

Teacher  Aides  and  Educational  Assistants,  Clerical 

2.22 

2 

55338 

Bookkeeping,  Accounting,  and  Auditing  Clerks 

1.93 

2 

53102 

Tellers 

1.92 

2 

49011 

Salespersons,  Retail 

1.85 

2 

55108 

Secretaries,  except  Legal  and  Medical 

1.82 

2 

49021 

Stock  Clerks,  Sales  Floor 

1.81 

2 

65008 

WaiterAVai  tress 

1.75 

2 

55305 

Receptionists  and  Information  Clerks 

1.74 

2 

55347 

General  Office  Clerks 

1.74 

2 

66008 

Nursing  Aides,  Orderlies,  and  Attendants 

1.72 

2 

49023 

Cashiers 

1.67 

2 

97111 

Bus  Drivers,  School 

1.32 

2 

85132 

Maintenance  Repairers,  General  Utility 

2.09 

3 

97102 

Truck  Drivers,  Heavy  and  Tractor-Trailer 

1.84 

3 

87902 

Earth  Drillers,  except  Oil  or  Gas 

1.70 

3 

67005 

Janitors  and  Cleaners,  except  Maids  and  Housekeeping  Cleaners 

1.52 

3 

65038 

Food  Preparation  Workers 

1.24 

3 

92974 

Packaging  and  Filling  Machine  Operators  and  Tenders 

1.23 

3 

O 

ERIC 
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Figure  14-13 

Twenty-Nine  Occupations  Clustered  on  Analysts'  Ratings  of  46  Basic  and  Cross-Functional  Skills  (Ward's 
Method) 


Occupation 

Code 

Occupation  Title 

Mean 
Across  46 
Skills 

Cluster 

19005 

General  Managers  and  Top  Executives 

4.02 

i 

15005 

Education  Administrators 

3.82 

i 

25105 

Computer  Programmers 

3.61 

i 

61005 

Police  and  Detective  Supervisors 

3.39 

i 

32902 

Medical  and  Clinical  Laboratory  Technologists 

3.20 

i 

32502 

Registered  Nurses 

3.05 

i 

51002 

First  Line  Supervisors  and  Managers,  Clerical/Admin. 

2.89 

i 

31305 

Teachers,  Elementary  School 

2.79 

i 

49008 

Sales  Representatives,  except  Scientific  and  Related  Products  or 
Services  and  Retail 

2.76 

i 

65026 

Cooks,  Restaurant 

1.94 

i 

63014 

Law  Enforcement  Officers 

2.55 

2 

66008 

Nursing  Aides,  Orderlies,  and  Attendants 

1.87 

2 

53905 

Teacher  Aides  and  Educational  Assistants,  Clerical 

1.59 

2 

55338 

Bookkeeping,  Accounting,  and  Auditing  Clerks 

1.57 

2 

55108 

Secretaries,  except  Legal  and  Medical 

1.53 

2 

49011 

Salespersons,  Retail 

1.48 

2 

55305 

Receptionists  and  Information  Clerks 

1.48 

2 

55347 

General  Office  Clerks 

1.46 

2 

49023 

Cashiers 

1.44 

2 — 

53102 

Tellers 

1.42 

2 

49021 

Stock  Clerks,  Sales  Floor 

1.36 

2 

65008 

Waiter/Waitress 

1.08 

2 

65038 

Food  Preparation  Workers 

.94 

2 

85132 

Maintenance  Repairers,  General  Utility 

2.13 

3 

97102 

Truck  Drivers,  Heavy  and  Tractor-Trailer 

1.58 

3 

87902 

Earth  Drillers,  except  Oil  or  Gas 

1.41 

3 

92974 

Packaging  and  Filling  Machine  Operators  and  Tenders 

1.34 

3 

67005 

Janitors  and  Cleaners,  except  Maids  and  Housekeeping  Cleaners 

1.31 

3 

97111 

Bus  Drivers,  School 

1.22 

3 

Figure  14-14 


Group  Avera 

« — j -rv  iu  unv/uunai  Giulia  1 rv  lumi- 

ee  Linkage") 

Occupation 

Code 

Occupation  Title 

Mean 
Across  46 
Skills 

Cluster 

19005 

General  Managers  and  Top  Executives 

4.02 

1 

15005 

Education  Administrators 

3.82 

1 

61005 

Police  and  Detective  Supervisors 

3.39 

1 

32902 

Medical  and  Clinical  Laboratory  Technologists 

3.20 

1 

32502 

Registered  Nurses 

3.05 

1 

51002 

First  Line  Supervisors  and  Managers,  Clerical/Admin. 

2,89 

1 

31305 

Teachers,  Elementary  School 

[ 2.79 

1 

49008 

Sales  Representatives,  except  Scientific  and  Related  Products  or 
Services  and  Retail 

2.76 

1 

63014 

Law  Enforcement  Officers 

2.55 

1 

65026 

Cooks,  Restaurant 

1.94 

1 

25105 

Computer  Programmers 

3.61 

2 

66008 

Nursing  Aides,  Orderlies,  and  Attendants 

1.87 

2 

53905 

Teacher  Aides  and  Educational  Assistants,  Clerical 

1.59 

2 

55338 

Bookkeeping,  Accounting,  and  Auditing  Clerks 

1.57 

2 

55108 

Secretaries,  except  Legal  and  Medical 

1.53 

2 

49011 

Salespersons,  Retail 

1.48 

2 

55305 

Receptionists  and  Information  Clerks 

1.48 

2 

55347 

General  Office  Clerks 

1.46 

2 

49023 

Cashiers 

1.44 

2 

53102 

Tellers 

1.42 

2 

49021 

Stock  Clerks,  Sales  Floor 

1.36 

2 

65008 

Waiter/Waitress 

1.08 

2 

65038 

Food  Preparation  Workers 

.94 

2 

85132 

Maintenance  Repairers,  General  Utility 

2.13 

3 

97102 

Truck  Drivers,  Heavy  and  Tractor-Trailer 

1.58 

3 

87902 

Earth  Drillers,  except  Oil  or  Gas 

1.41 

3 

92974 

Packaging  and  Filling  Machine  Operators  and  Tenders 

1.34 

3 

67005 

Janitors  and  Cleaners,  except  Maids  and  Housekeeping  Cleaners 

1.31 

3 

97111 

Bus  Drivers,  School 

1.22 

3 

Figure  14-15 

Twenty-Nine  Occupations  Clustered  on  Analysts*  Ratings  of  42  Generalized  Work  Activities  and  46  Basic 
and  Cross-Functional  Skills  (Ward's  Method) 


Occupation 

Code 

Occupation  Title 

Mean 
Across  42 
GW  As  and 
46  Skills 

Cluster 

19005 

General  Managers  and  Top  Executives 

4.09 

1 

15005 

Education  Administrators 

3.85 

1 

61005 

Police  and  Detective  Supervisors 

3.58 

1 

25105 

Computer  Programmers 

3.45 

1 

32902 

Medical  and  Clinical  Laboratory  Technologists 

3.20 

1 

32502 

Registered  Nurses 

3.19 

1 

31305 

Teachers,  Elementary  School 

2.96 

1 

51002 

First  Line  Supervisors  and  Managers,  Clerical/Admin. 

2.89 

1 

65026 

Cooks,  Restaurant 

2.19 

1 

63014 

Law  Enforcement  Officers 

2.80 

2 

49008 

Sales  Representatives,  except  Scientific  and  Related  Products  or 

2.69 

2 

53905 

Teacher  Aides  and  Educational  Assistants,  Clerical 

1.90 

2 

66008 

Nursing  Aides,  Orderlies,  and  Attendants 

1.79 

2 

55338 

Bookkeeping,  Accounting,  and  Auditing  Clerks 

1.75 

2 

55108 

Secretaries,  except  Legal  and  Medical 

1.68 

2 

53102 

Tellers 

1.67 

2 

49011 

Salespersons,  Retail 

1.67 

2 

55305 

Receptionists  and  Information  Clerks 

1.61 

2 

55347 

General  Office  Clerks 

1.60 

2 

49021 

Stock  Clerks,  Sales  Floor 

1.58 

2 

49023 

Cashiers 

1.56 

2 

65008 

Waiter/Waitress 

1.42 

2 

85132 

Maintenance  Repairers,  General  Utility 

2.11 

3 

97102 

Truck  Drivers,  Heavy  and  Tractor-Trailer 

1.71 

3 

87902 

Earth  Drillers,  except  Oil  or  Gas 

1.56 

3 

67005 

Janitors  and  Cleaners,  except  Maids  and  Housekeeping  Cleaners 

1.41 

3 

92974 

Packaging  and  Filling  Machine  Operators  and  Tenders 

1.29 

3 

97111 

Bus  Drivers,  School 

1.27 

3 

65038 

Food  Preparation  Workers 

1.09 

3 

Figure  14-16 

Twenty-Nine  Occupations  Clustered  on  Analysts’  Ratings  of  42  Generalized  Work  Activates  and  46  Basic 
and  Cross-Functional  Skills  (Within-Group  Average  Linkage! 


Occupation 

Code 

Occupation  Title 

Mean  Across 
42  GW  As 
and  46  Skills 

Cluster 

19005 

General  Managers  and  Top  Executives 

4.09 

1 

15005 

Education  Administrators 

3.85 

1 

61005 

Police  and  Detective  Supervisors 

3.58 

1 

32902 

Medical  and  Clinical  Laboratory  Technologists 

3.20 

1 

32502 

Registered  Nurses 

3.19 

1 

31305 

Teachers,  Elementary  School 

2.96 

1 

51002 

First  Line  Supervisors  and  Managers,  Clerical/ Admin. 

2.89 

1 

49008 

Sales  Representatives,  except  Scientific  and  Related  Products  or 

2.69 

1 

65026 

Cooks,  Restaurant 

2.19 

1 

63014 

Law  Enforcement  Officers 

2.80 

2 

53905 

Teacher  Aides  and  Educational  Assistants,  Clerical 

1.90 

2 

66008 

Nursing  Aides,  Orderlies,  and  Attendants 

1.79 

2 

55338 

Bookkeeping,  Accounting,  and  Auditing  Clerks 

1.75 

2 

55108 

Secretaries,  except  Legal  and  Medical 

1.68 

2 

53102 

Tellers 

1.67 

2 

4901 1 

Salespersons,  Retail 

1.67 

2 

55305 

Receptionists  and  Information  Clerks 

1.61 

2 

55347 

General  Office  Clerks 

1.60 

2 

49021 

Stock  Clerks,  Sales  Floor 

1.58 

2 

49023 

Cashiers 

1.56 

2 

65008  H 

Waiter/Waitress 

1.42 

2 

65038 

Food  Preparation  Workers 

1.09 

2 

— 

25105 

Computer  Programmers 

3.45 

3 

85132 

Maintenance  Repairers,  General  Utility 

2.11 

4 

97102 

Truck  Drivers,  Heavy  and  Tractor-Trailer 

1.71 

4 

87902 

Earth  Drillers,  except  Oil  or  Gas 

1.56 

4 

67005 

Janitors  and  Cleaners,  except  Maids  and  Housekeeping  Cleaners 

1.41 

4 

92974 

Packaging  and  Filling  Machine  Operators  and  Tenders 

1.29 

4 

97111 

Bus  Drivers,  School 

1.27 

4 

1297 


Figure  14-17 

Twenty-Nine  Occupations  Clustered  First  on  Analysts*  Ratings  of  42  Generalized  Work  Activities:  Second 
on  46  Basic  and  Cross-Functional  Skills  (Ward’s  Method) 


Occupation 

Code 

Occupation  Title 

Mean 
Across  46 
Skills 

Cluster 

19005 

General  Managers!  and  Top  Executives 

4.02 

i 

15005 

Education  Administrators 

3.82 

i 

61005 

Police  and  Detective  Supervisors 

3.39 

i 

51002 

First  Line  Supervisors  and  Managers,  Clerical/Admin. 

2.89 

i 

32902 

Medical  and  Clinical  Laboratory  Technologists 

3.20 

2 

32502 

Registered  Nurses 

3.05 

2 

31305 

Teachers,  Elementary  School 

2.79 

2 

65026 

Cooks,  Restaurant 

1.94 

2 

53905 

Teacher  Aides  and  Educational  Assistants,  Clerical 

1.59 

2 

25105 

Computer  Programmers 

3.61 

3 

55338 

Bookkeeping,  Accounting,  and  Auditing  Clerks 

1.57 

4 

63014 

Law  Enforcement  Officers 

2.55 

5 

66008 

Nursing  Aides,  Orderlies,  and  Attendants 

1.87 

5 

49008 

Sales  Representatives,  except  Scientific  and  Related  Products  or 
Services  and  Retail 

2.76 

6 

55108 

Secretaries,  except  Legal  and  Medical 

1.53 

7 

49011 

Salespersons,  Retail 

1.48 

7 

55305 

Receptionists  and  Information  Clerks 

1.48 

7 

55347 

General  Office  Clerks 

1.46 

7 

49023 

Cashiers 

1.44 

7 

1 53102 

Tellers 

1.42 

7 

49021 

Stock  Clerks,  Sales  Floor 

1.36 

7 

65008 

Waiter/Wai  tress 

1.08 

7 

97111 

Bus  Drivers,  School 

1.22 

8 

♦ 

65038 

Food  Preparation  Workers 

.94 

9 

97102 

Truck  Drivers,  Heavy  and  Tractor-Trailer 

1.58 

10 

87902 

Earth  Drillers,  except  Oil  or  Gas 

1.41 

11 

92974 

Packaging  and  Filling  Machine  Operators  and  Tenders 

1.34 

11 

85132 

Maintenance  Repairers,  General  Utility 

2.13 

12 

67005 

Janitors  and  Cleaners,  except  Maids  and  Housekeeping  Cleaners 

1.31 

12 

1298 
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Figure  14-18 

Twenty-Nine  Occupations* Clustered  First  on  Analysts'  Ratings  of  42  Generalized  Work  Activities;  Second 
on  46  Basic  and  Cross-Functional  Skills  (Within-Group  Average  Linkage) 


Occupation 

Code 

Occupation  Title 

Mean 
Across  46 
Skills 

Cluster 

19005 

General  Managers  and  Top  Executives 

4.02 

i 

15005 

Education  Administrators 

3.82 

i 

61005 

Police  and  Detective  Supervisors 

3.39 

i 

51002 

First  Line  Supervisors  and  Managers,  Clerical/Admin. 

2.89 

i 

32902 

Medical  and  Clinical  Laboratory  Technologists 

3.20 

2 

32502 

Registered  Nurses 

3.05 

2 

31305 

Teachers,  Elementary  School 

2.79  1 

2 

65026 

Cooks,  Restaurant 

1.94 

2 

25105 

Computer  Programmers 

3.61 

3 

63014 

Law  Enforcement  Officers 

2.55 

4 

66008 

Nursing  Aides,  Orderlies,  and  Attendants 

1.87 

4 

53905 

Teacher  Aides  and  Educational  Assistants,  Clerical 

1.59 

4 

55338 

Bookkeeping,  Accounting,  and  Auditing  Clerks 

1.57 

4 

55108 

Secretaries,  except  Legal  and  Medical 

1.53 

4 

49011 

Salespersons,  Retail 

1.48 

4 

55305 

Receptionists  and  Information  Clerks 

1.48 

4 

55347 

General  Office  Clerks 

1.46 

4 

49023 

Cashiers 

1.44 

4 

53102 

Tellers 

1.42 

4 

49021 

Stock  Clerks,  Sales  Floor 

1.36 

4 

65008 

Waiter/Waitress 

1.08 

4 

49008 

Sales  Representatives,  except  Scientific  and  Related  Products  or 
Services  and  Retail 

2.76 

5 

97111 

Bus  Drivers,  School 

1.22 

6 

65038 

Food  Preparation  Workers 

.94 

7 

85132 

Maintenance  Repairers,  General  Utility 

2.13 

8 

97102 

Truck  Drivers,  Heavy  and  Tractor-Trailer 

1.58 

8 

87902 

Earth  Drillers,  except  Oil  or  Gas 

1.41 

8 

92974 

Packaging  and  Filling  Machine  Operators  and  Tenders 

1.34 

8 

67005 

Janitors  and  Cleaners,  except  Maids  and  Housekeeping  Cleaners 

1.31 

8 

1299 
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Figure  14-19 

Twenty-Nine  Occupations  Clustered  First  on  Analysts*  Ratings  of  46  Basic  and  Cross-FunctionaLSkills; 
Second  on  42  Generalized  Work  Activitites  fW ard’s  Method)  


Occupation 

Code 

Occupation  Title 

Mean 
Across  42 
GWAs 

Cluster 

19005 

General  Managers  and  Top  Executives 

4.16 

i 

15005 

Education  Administrators 

3.88 

i 

61005 

Police  and  Detective  Supervisors 

3.77 

i 

51002 

First  Line  Supervisors  and  Managers,  Clerical/Admin. 

2.90 

i 

32502 

Registered  Nurses 

3.33 

2 

32902 

Medical  and  Clinical  Laboratory  Technologists 

3.19 

2 

31305 

Teachers,  Elementary  School 

3.13 

2 

49008 

Sales  Representatives,  except  Scientific  and  Related  Products  or 
Services  and  Retail 

2.62 

3 

25105 

Computer  Programmers 

3.30 

4 

65026 

Cooks,  Restaurant 

2.44 

5 

63014 

Law  Enforcement  Officers 

3.05 

6 

49011 

Salespersons,  Retail 

1.85 

6 

49021 

Stock  Clerks,  Sales  Floor 

1.81 

6 

65008 

Waiter/Waitress 

1.75 

6 

53905 

Teacher  Aides  and  Educational  Assistants,  Clerical 

2.22 

7 

55338  ' 

Bookkeeping,  Accounting,  and  Auditing  Clerks 

1.93 

7 

53102 

Tellers 

1.92 

7 

55108 

Secretaries,  except  Legal  and  Medical 

1.82 

7 

55305 

Receptionists  and  Information  Clerks 

1.74 

7 

55347 

General  Office  Clerks 

1.74 

7 

49023 

Cashiers 

1.67 

7 

- 

66008 

Nursing  Aides,  Orderlies,  and  Attendants 

1.72 

8 

65038 

Food  Preparation  Workers 

1.24 

8 

85132 

Maintenance  Repairers,  General  Utility 

2.09 

9 

97102 

Truck  Drivers,  Heavy  and  Tractor-Trailer 

1.84 

9 

87902 

Earth  Drillers,  except  Oil  or  Gas 

1.70  ' 

1 9 

67005 

Janitors  and  Cleaners,  except  Maids  and  Housekeeping  Cleaners 

1.52 

9 

92974 

Packaging  and  Filling  Machine  Operators  and  Tenders 

1.23 

9 

97111 

Bus  Drivers,  School 

1.32 

10 

1300 


Figure  14-20 


l WU>  1 1 L V ” 1 

Second  o 

line  wnupaiiuns  Clustered  rirst  on  Analysts'  Katines  ot  46  Basic  and  Cross-Functional  Skills- 
42  Generalized  Work  Activitites  fWithin-GrouDS  Averaee  Linkage^ 

Occupa- 

tion 

Code 

Occupation  Title 

Mean 

Across 

42 

GWAs 

Cluster 

19005 

General  Managers  and  Top  Executives 

4.16 

i 

15005 

Education  Administrators 

3.88 

i 

61005 

Police  and  Detective  Supervisors 

3.77 

i 

51002 

First  Line  Supervisors  and  Managers,  Clerical/Admin. 

2.90 

i 

63014 

Law  Enforcement  Officers 

3.05 

2 

32502 

Registered  Nurses 

3.33 

3 

31305 

Teachers,  Elementary  School 

3.13 

3 

49008 

Sales  Representatives,  except  Scientific  and  Related  Products  or  Services 
and  Retail 

2.62 

4 

32902 

Medical  and  Clinical  Laboratory  Technologists 

3.19 

5 

65026 

Cooks,  Restaurant 

2.44 

6 

25105 

Computer  Programmers 

3.30 

7 

53905 

Teacher  Aides  and  Educational  Assistants,  Clerical 

2.22 

8 

55338 

Bookkeeping,  Accounting,  and  Auditing  Clerks 

1.93 

8 

53102 

Tellers 

1.92 

8 

55108 

Secretaries,  except  Legal  and  Medical 

1.82 

8 

55305 

Receptionists  and  Information  Clerks 

1.74 

8 

55347 

General  Office  Clerks 

1.74 

8 

49023 

Cashiers 

1.67 

8 

49011 

Salespersons,  Retail 

1.85 

9 

49021 

Stock  Clerks,  Sales  Floor 

1.81 

9 

65008 

Waiter/Waitress 

1.75 

9 

66008 

Nursing  Aides,  Orderlies,  and  Attendants 

1.72 

10 

65038 

Food  Preparation  Workers 

1.24 

10 

85132 

Maintenance  Repairers,  General  Utility 

2.09 

11 

97102 

Truck  Drivers,  Heavy  and  Tractor-Trailer 

1.84 

11 

87902 

Earth  Drillers,  except  Oil  or  Gas 

1.70 

11 

67005 

Janitors  and  Cleaners,  except  Maids  and  Housekeeping  Cleaners 

1.52 

11 

92974 

Packaging  and  Filling  Machine  Operators  and  Tenders 

1.23 

11 

97111 

Bus  Drivers,  School 

1.32 

12 

O 
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Figure  14-21 

Final  Five  Clusters  from  1 122  Occupations  Clustered  First  on  Analysts'  Ratings  of  46  Basic  and  Cross- 
Functional  Skills;  Second  on  42  Generalized  Work  Activities  (Ward’s  Method) 


Occupation 

Code 

Occupation  Title 

Mean 
Across  42 
GWAs 

Cluster 

24502B 

Artificial  Breeding  Technicians 

1.30 

177 

13017A 

Engineering  Managers 

4.25 

178 

24308A 

Biochemists 

3.04 

179 

24308C 

Biophysicists 

2.94 

179 

24199C 

Materials  Scientists 

2.66 

179 

27199H 

Archeologists 

2.47 

179 

24 199 A 

Geographers 

2.12 

179 

24308F 

Geneticists 

3.58 

179 

24308G 

Physiologists  and  Cytologists 

3.35 

179 

24308B 

Biologists 

3.34 

179 

24308J 

Toxicologists 

3.10 

179 

24308E 

Microbiologists 

3.07 

179 

24308D 

Botanists 

2.91 

179 

2191  IP 

Coroners 

2.69 

179 

24308H 

Zoologists 

2.40 

179 

22127 

Computer  Engineers 

3.56 

180 

25102 

Systems  Analysts,  Electronic  Data  Processing 

3.30 

180 

25105 

Computer  Programmers 

3.30 

180 

2 1999 A 

Computer  Security  Specialists 

2.91 

180 

225 14B 

Electronic  Drafters 

2.60 

180 

25 103 A 

Data  Base  Administrators 

2.35 

180 

25104 

Computer  Support  Specialists 

2.63 

180 

25 199 A 

Data  Communications  Analysts 

. 2.57 

180 

49002 

Sales  Engineers 

3.37 

181 

49005D 

Sales  Representatives,  Mechanical  Equipment  and  Supplies 

2.36 

181 

49005F 

Sales  Representatives,  Medical 

2.20 

181 

49005B 

Sales  Representatives,  Chemical  and  Pharmaceutical 

2.03 

181 

49005A 

Sales  Representatives,  Agricultural 

1.93 

181 

49005C 

Sales  Representatives,  Electrical/Electronic 

1.85 

181 

49026 

Telemarketers,  Door-To-Door  Sales  Workers,  News  and  Street 
Vendors,  and  Related  Workers 

1.84 

181 

49005G 

Sales  Representatives,  Instruments 

1.80 

181 

'V 


Table  14-1 

Criteria  Estimates  for  Cluster  Solutions  Across  Study  Designs 


Incumbent  Ratings 
29  Occupations 

Variable  sets  Ward's  Avg.  Linkage 


GWAs  only 

# of  clusters  4 3 

single  occupation  clusters  0 0 

% reclassification  96.60  100.00 

simple  structure  100.00  100.00 

MSE  10.63  17.04 

effect  size  0.58  0.29 

Skills  only 

# of  clusters  5 6 

single  occupation  clusters  1 3 

% reclassification  100.00  89.70 

simple  structure  100.00  100.00 

MSE  25.54  35.35 

effect  size  0.49  0.32 

GWAs  & Skills 

# of  clusters  6 9 

single  occupation  clusters  1 7 

% reclassification  100.00  96.60 

Variable  sets 

simple  structure  100.00  100.00 

MSE  28.75  12.13 

effect  size  0.59  0.85 

GWAs,  Skills  (2-stage) 

# of  clusters  11  -9 

single  occupation  clusters  4 3 

% reclassification  100.00  100.00 

simple  structure  100.00  100.00 

MSE  19.24  21.73 

effect  size  0.79  0.73 

Skills,  GWAs  (2-stage) 

# of  clusters  11  12 

single  occupation  clusters  5 7 

% reclassification  100.00  96.60 

simple  structure  100.00  100.00 

MSE  25.56  13.85 

effect  size 0.72  0.86 


Analysts  Ratings 

29  Occupations  1122  Occupations 


Wards 

Avg.  Linkage 

Ward’s 

Avg.  Linkage 

3 

3 

41 

41 

0 

0 

0 

0 

96.60 

93.10 

83.10 

76.60 

100.00 

100.00 

100.00 

100.00 

12.63 

8.44 

5.84 

6.30 

0.60 

0.74 

0.77 

0.75 

3 

3 

45 

45 

0 

0 

0 

0 

100.00 

100.00 

82.70 

78.40 

100.00 

100.00 

100.00 

100.00 

9.96 

15.72 

7.96 

8.50 

0.76 

0.62 

0.73 

0.71 

3 

4 

66 

66 

0 

1 

0 

0 

89.70 

89.70 

88.40 

77.50 

100.00 

100.00 

100.00 

100.00 

18.49 

17.85 

9.06 

10.89 

0.74 

0.76 

0.83 

0.79 

12 

8 

176 

149 

6 

4 

58 

47 

100.00 

100.00 

90.10 

86.40 

100.00 

100.00 

100.00 

100.00 

16.00 

13.63 

8.53 

9.45 

0.85 

0.85 

0.86 

0.84 

10 

12 

181 

160 

4 

6 

67 

48 

100.00 

100.00 

89.80 

87.60 

100.00 

100.00 

100.00 

100.00 

12.81 

8.27 

8.69 

9.33 

0.87 

0.92 

0.85 

0.84 

Note,  Avg.  = Average;  GWAs  = generalized  work  activities;  skills  = basic  and  cross-functional  skills;  MSE  = mean  squared  error. 
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Table  14-2 

Convergence  (Cramer’s  V)  of  Cluster  Solutions  Based  on  Different  Variable  Sets 

Incumbent  Ratines  Analysts  Ratines 

29  Occupations  29  Occupations  U22  Occupation? 


Avg.  Avg.  Avg. 


Variable  Set  Comparisons  . 

Ward’s 

Linkage 

Ward’s 

Linkage 

Ward’s 

Linkage 

GWAs  v.  Skills 

0.59 

0.61 

0.75 

0.75 

0.54 

0.52 

GWAs  v.  Combined 

0.80 

0.75 

0.75 

0.84 

0.72 

0.72 

Skills  v.  Combined 

0.81 

0.93 

0.90 

0.96 

0.70 

0.68 

GWAs  v.  Skills, 
GWAs  (2-stage) 

0.80 

0.88 

0.90 

0.97 

0.78 

0.75 

Skills  v.  GWAs, 
Skills  (2-stage) 

0.97 

0.91 

0.96 

0.96 

0.75 

0.69 

Combined  v. 
GWAs,  Skills  (2- 
stage) 

0.91 

0.78  . 

0.96 

1.00 

0.80 

0.76 

Combined  v.  Skills, 
GWAs  (2-stage) 

0.88 

0.94 

0.97 

1.00 

0.77 

0.78 

Skills,  GWAs  (2- 
stage)  v.  GWAs, 
Skills  (2-stage) 

0.86 

0.91 

0.85 

0.96 

0.64 

0.64 

Note.  Avg.  = average;  GW  As  = generalized  work  activities;  skills  = basic  & cross  functional  skills; 
combined  = skills  & GW  As  together,  v.  = versus. 
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Table  14-3 

Convergence  (Cramer's  V)  of  Cluster  Solutions  Based  on  Ward's  and  Within-gmnps 
Average  Linkage  Methods 


Variable  Sets 

Incumbent 

Ratings 

29  OccuDations 

Analysts  Ratings 

29  Occupations  1 122  OccuDations 

GWAs  only 

0.78 

0.92 

0.73 

Skills  only 

0.83 

0.92 

0.70 

GWAs  & Skills 

0.80 

0.90 

0.68 

GWAs,  Skills  (2-stage) 

0.96 

0.98 

0.74 

Skills,  GWAs  (2-stage) 

0.85 

1.00 

0.74 

Mote^  GWAs  = generalized  work  activities;  skills  = basic  & cross  functional  skills. 
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Table  14-4 

Convergence  ( Cramer's  VI  of  Cluster  Solutions  Based  on  Ratings 
From  Incumbents  and  Analysts  on  the  29  Occupation  Database 


Variable  Sets 

Ward's 

Avg.  Linkage 

GW  As  only 

0.64 

0.50 

Skills  only 

0.79 

0.62 

GWAs  & Skills 

0.91 

0.93 

GWAs,  Skills  (2-stage) 

0.77 

0.78 

Skills,  GWAs  (2-stage) 

0.81 

0.80 

Note.  GWAs  = generalized  work  activities;  skills 

= basic  & cross  functional  skills. 
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Chapter  15 

Issues  in  0*NET  Applications 


Walter  C.  Borman 
Mary  Ann  Hanson 
U.  Christean  Kubisiak 

Personnel  Decisions  Research  Institutes,  Incorporated 


The  0*NET  database  should  be  useful  for  many  applications.  In  addition,  there  are 
several  user  groups  who  should  benefit  from  0*NET.  The  topic  of  applications  and  potential 
users  has  been  discussed  previously  in  Peterson,  Mumford,  Borman,  Jeanneret,  and  Fleishman 
(1995).  However,  now  that  data  have  been  gathered  on  the  0*NET  descriptors  for  at  least  some 
occupations  and  initial  analyses  of  these  data  have  been  accomplished,  we  are  able  to  discuss 
issues  regarding  applications  with  more  knowledge  about  the  kinds  of  information  actually  likely 
to  be  available  in  the  0*NET  database. 

The  present  chapter  begins  with  a general  discussion  of  the  implications  of  some  of  the 
analysis  results  for  likely  applications  of  the  0*NET.  Next,  we  briefly  review  some  specific 
programs  that  could  benefit  from  the  0*NET  and  discuss  implications  of  the  available  results  for 
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these  initiatives.  Issues  in  using  the  0*NET  data  to  generate  occupation  descriptions  are  then 

discussed  in  some  detail.  Finally,  we  provide  an  example  of  one  such  application  of  the  0*NET 
database. 

Implications  of  the  Analysis  Results  for  0*NET  Applications 
As  described  in  Peterson  et  al.  (1995),  the  0*NET  should  provide  important  job  and 
occupational  information  for  such  users  as  employers,  job  seekers,  career  explorers,  vocational 
counselors,  and  researchers  interested  in  jobs  and  work.  General  applications  of  this  information 
include  employers  selecting  workers,  individuals  evaluating  jobs  in  relation  to  what  they  are 
qualified  for  and  where  training  might  be  needed,  students  and  others  exploring  career  options, 
dislocated  workers  trying  to  identify  new  job  opportunities,  educators  and  trainers  developing 
programs  to  deliver  the  most  relevant  training,  and  researchers  learning  about  similarities  and 
differences  between  occupations  in  terms  of  their  skill,  ability,  etc.,  requirements. 

In  this  section,  selected  analysis  results  are  reviewed  with  a focus  toward  examining  their 
impact  on  0*NET  applications.  First,  the  reliability  results  have  implications  for  many  if  not  all 
applications.  The  fact  that  interrater  agreement  was  generally  high  for  the  descriptors  means  that 
profiles  for  occupations  have  meaning  in  the  sense  that  they  are  stable  representations  of  these 
occupations’  requirements.  For  example,  when  a search  is  initiated  for  the  highest  level  skill 
requirements  for  a particular  occupation,  the  answer  is  likely  to  be  reliable  and  replicable. 

Another  positive  finding  is  that  almost  all  of  the  descriptors  have  variance  across 
occupations.  Only  for  the  work  style  and  occupational  value  domains  were  the  obtained  data 
somewhat  restricted.  For  example,  Dependability  (Work  Styles  Descriptor  #12)  scores  were 
uniformly  high  for  the  35  occupations  in  the  sample.  However,  for  most  of  the  other  descriptors 
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across  the  domains,  variance  in  the  scores  across  occupations  provides  differentiation  that  is  a 
prerequisite  for  many  applications. 

The  discriminant  function  analysis  demonstrated  this  differentiation  in  a more  definitive 
way.  Results  of  these  analyses  showed  that  almost  all  descriptors  contributed  to  differentiation 
between  occupations.  The  only  exception  to  this  finding  was,  once  again,  some  of  the  work  style 
and  occupational  value  descriptors  (e.g..  Dependability).  The  fact  that  descriptor  scores  vary 
substantially  across  occupations  is  very  important,  especially  for  exploring  career  options  relative 
to  several  occupations  and  learning  about  similarities  and  differences  between  jobs/occupations. 

Some  of  the  intended  applications  of  the  0*NET  information  require  the  hierarchical 
feature  of  the  content  model.  Consider,  for  example,  a student  exploring  the  skill  requirements  of 
several  occupations  with  which  he/she  is  unfamiliar.  As  a first  sweep,  the  student  may  want  to 
enter  0*NET  at  the  higher  level  of  the  skills  taxonomy  and  then  later  explore  more  in  depth  at 
the  lower  level  after  eliminating  some  occupations  from  consideration.  A strong  case  could  be 
made  for  using  the  hierarchical  models  in  this  manner  if  the  exploratory  factor  analysis  results 
supported  the  hierarchies;  i.e.,  the  empirical  factors  were  aligned  with  the  higher-order 
categories.  This  was  largely  true  of  the  GW  As  where  the  3-factor  solution  represented  two  of  the 
four  highest-order  categories  and  the  third  factor  reflected  a composite  of  the  other  two 
categories.  In  many  of  the  other  domains,  the  factor  structure  did  not  entirely  reproduce  the 
higher-order  category  systems,  but  for  most  domains  the  results  are  not  inconsistent  with  the 
content  model.  That  is,  descriptors  from  the  same  content  model  category  generally  load  most 
highly  on  the  same  factor.  We  must  keep  in  mind  that  these  factor  analysis  results  are  based  on 
very  small  numbers  of  occupations  (Ns  ranging  from  30  to  37  for  each  domain).  A larger  N may 
yield  factor  solutions  more  similar  to  the  proposed  hierarchical  systems.  In  addition,  many  of  the 
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domains  in  the  content  model  contain  more  specific,  second-order  categories,  and  a much  larger 
N is  needed  to  test  these  more  detailed  taxonomies.  Finally,  the  higher-order  taxonomies  for 
some  domains  are  not  solely  intended  to  capture  past  empirical  findings,  but  to  incorporate 
conceptual  and  developmental  notions,  notably  in  the  skills  domain. 

An  even  stronger  case  for  these  hierarchies  could  be  made  by  conducting  confirmatory 
factor  analyses  (CFA).  Unfortunately,  CFA  is  quite  sensitive  to  sample  size,  and,  again,  the  N 
here  is  very  low.  An  attempt  was  made  to  conduct  CFA  for  the  content  model  hierarchies  in 
several  domains,  and  these  analyses  generally  did  not  converge,  probably  due  to  the  small 
number  of  occupations  currently  available.  When  the  number  of  occupations  represented  in  the 
sample  is  larger,  CFA  can  be  used  within  each  domain  to  test  the  hypothesis  that  lower-order 
descriptors  load  on  the  intended  higher-order  categories  and  not  on  other  higher-order  categories. 
In  addition,  larger  sample  sizes  should  allow  tests  of  the  more  detailed,  second-order  categories. 

In  order  to  provide  more  targeted,  although  preliminary,  empirical  evidence  concerning 
support  for  the  content  model,  we  conducted  somewhat  simplistic  confirmatory  analyses.  For 
each  content  domain,  we  intercorrelated  all  of  the  descriptors  across  occupations  and  identified 
those  correlations  in  this  matrix  that  involved  descriptors  from  the  same  second-order  category  in 
the  content  model  hierarchy.  These  within-category  correlations  were  then  converted  using 
Fisher’s  r to  z transformation,  averaged,  and  the  resulting  average  z-scores  were  converted  back 
to  standard  correlation  coefficients.  This  mean  correlation  within  second-order  categories  was 
then  compared  with  the  mean  correlation  across  second-order  categories.  The  latter  correlation 
was  simply  the  average  (again  using  an  r to  z transformation)  of  all  of  the  remaining  correlations, 
that  is,  those  that  involved  descriptors  from  different  second-order  categories.  Table  15-1 
presents  the  results  of  these  analyses.  In  addition  to  the  mean  correlations,  this  table  also  shows 
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the  minimum  and  maximum  correlation  obtained,  both  within  and  between  categories,  for  each 
domain.  This  confirmatory  analysis  was  not  attempted  for  organizational  context,  work  context, 
or  training,  licensure,  and  experience,  because  the  content  model  hierarchies  in  these  areas  are 
based  on  conceptual  similarities  among  the  descriptors  rather  than  expected  empirical 
relationships.  _ 

Table  15-1  shows  that,  overall,  the  analyses  support  our  hierarchical  models  in  almost  all 
domains.  Correlations  between  descriptors  within  higher  level  categories  were  substantially 
higher  than  correlations  between  descriptors  across  higher  level  categories,  except  for  the  skills 
and  occupational  values  domains.  In  the  latter  two  cases  these  correlations  were  close  to  equal. 
The  largest  difference  is  for  abilities.  The  descriptors  in  the  ability  domain  have  a very  strong 
base  of  past  research  and  theory.  In  addition,  the  conceptual  differences  between  the  various 
second-order  categories  in  the  ability  domain  are  substantial.  These  categories  include  Verbal 
Abilities;  Fine  Manipulative  Abilities;  and  Flexibility,  Balance  and  Coordination.  Table  15-1 
also  shows  support  for  the  hierarchical  structure  of  the  second-order  categories  for  generalized 
work  activities,  work  styles,  and  knowledges. 

For  the  skills  domain,  the  average  within  and  between  correlations  are  virtually  identical. 
This  could  be  due  to  the  fact  that  this  domain  is  not  as  clearly  hierarchically  structured  as  the 
other  domains,  as  previously  mentioned.  There  is  a great  deal  of  interdependence  among  skills 
that  are  assigned  to  different  second-order  categories.  Table  15-1  does  show  that  the 
intercorrelations  among  skills  are  generally  high,  both  within  and  between  second-order 
categories,  so  the  aggregation  of  skills  to  a higher  level  is  clearly  supported,  but  it  may  be  that  we 
need  to.re-examine  the  current  organization  of  skills,  or  otherwise  carefully  consider  how  we 
form  aggregates  here.  Results  for  occupational  values  are  also  not  supportive  of  the  a priori 
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taxonomy.  The  0*NET  occupational  values  taxonomy  is  based  on  previous  factor  analytic  work 
in  this  area  that  focused  on  worker  traits,  so  it  is  not  completely  surprising  that  the  factor 
structure  appears  to  differ  for  the  0*NET  data,  where  the  focus  is  on  occupational 
characteristics/requirements. 

Although  in  most  domains  the  mean  within-higher-order  category  correlations  were 
substantially  higher  than  those  across  higher-order  categories,  the  within-category  correlations 
are  not  uniformly  high.  For  example.  Repairing  and  Maintaining  Mechanical  Equipment  and 
Interacting  with  Computers,  both  part  of  the  Performing  Complex/Technical  Activities  category 
m the  GWA  domain,  correlate  -.26.  Accordingly,  it  does  not  appear  appropriate  to  have  these  two 
descriptors  in  the  same  higher-order  category.  Perhaps  descriptor  membership  in  the  higher-order 
categories  can  be  reconfigured  somewhat  to  conform  better  to  the  correlation  matrix,  Again, 
however,  we  hesitate  to  make  major  changes  to  the  content  model  based  on  data  from  just  29 
occupations.  Given  the  general  support  for  the  content  model  hierarchies  that  has  been  obtained 
in  the  current  analyses,  the  content  model  seems  to  provide  a good  starting  point  for 
implementing  a hierarchical  structure  in  the  0*NET  database.  It  is  likely  that  this  system  can  and 
should  be  refined  when  data  on  more  occupations  become  available. 

Relatively  high  empirical  correlations  of  descriptor  scores  within  higher-order  categories 
is  especially  important  for  aggregation  strategies  that  involve  computing  sums  or  means  across 
descriptors.  This  may  not  be  the  best  strategy  for  every  domain.  Certainly,  for  some  categories  in 
skills,  this  would  not  seem  appropriate  (e.g.,  resource  management).  If  the  aggregation  strategy 
selected  is  “single  highest  descriptor  within  a category,”  for  example,  correlations  are  less 
important.  These  issues  are  carefully  considered  below. 
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Many  of  the  applications  of  0*NET  will  involve  data  from  multiple  domains  (e.g., 
abilities  and  skills).  For  example,  in  selection  applications,  matches  between  person  attributes 
and  the  job  requirements  will  likely  need  to  include  skills,  abilities,  and  perhaps  knowledges  and 
work  styles.  Fortunately,  the  across  domain  analysis  results  show  considerable  coherence  as  well, 
even  with  the  small  N.  Accordingly,  information  about  individual  occupations  across  domains 
should  make  sense,  and  applications  requiring  the  use  of  more  than  one  domain  should  thus  be 
well  served. 

The  very  high  correlations  between  the  level  and  importance  rating  scales  may  have 
implications  for  some  0*NET  applications.  Essentially,  data  from  one  scale— level  or 
importance— provides  quite  definitive  information  about  the  other,  although  this  varies  somewhat 
across  occupations.  This  allows  the  options  of  foregoing  either  level  or  importance  at  some  point 
in  the  future,  but  not  without  some  loss  of  information. 

Overall  then,  the  data  analysis  results  are  supportive  of  most  foreseeable  applications  of 
0*NET.  The  reliability  findings  are  probably  most  important,  suggesting  that  information  about 
occupations  on  the  0*NET  reflect  stable,  accurate  data.  Discriminant  function  analysis  results 
are  also  very  important.  They  indicate  that  occupations  can  be  differentiated  using  0*NET 
descriptors,  which  in  turn  means  that  applications  requiring  identification  of  differences  between 
occupations  in  skill,  ability,  GW  A,  etc.,  requirements  should  be  successful.  Some  questions 
remain,  however,  concerning  the  optimal  hierarchical  structure  of  the  content  model  domains. 
There  is  some  empirical  support  for  the  hypothesized  structures  within  domains,  but  CFA  results 
using  larger  numbers  of  occupations  confirming  these  hierarchical  systems  would  provide  better 
evidence.  It  is  likely  that  some  refinement  of  these  structures  will  be  necessary  in  the  future,  but 
the  present  structures  seem  sufficient  for  the  near  future. 
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Highlights  of  Implications  for  Specific  Applications 
Another  way  to  discuss  the  implications  of  the  analysis  results  for  applications  associated 
with  0*NET  is  to  review  several  general  areas  of  policy  initiatives  and  the  occupational  data 
needs  for  each  of  these  initiatives.  For  each  of  these  initiatives,  the  relevant  aspects  of  the 
0*NET  are  described,  and  the  implications  of  analysis  results  available  to  date  are  discussed. 

Educational  Policy  and  Skill  Standards 

A need  for  this  area  is  an  occupational  framework  for  developing  skills  standards.  The 
content  model  is  clearly  responsive  to  this  need.  A comprehensive  set  of  skill  descriptors  are 
included  for  describing  occupational  skill  requirements,  as  well  as  descriptors  depicting  a wide 
variety  of  work  activities.  Data  collected  using  these  descriptors  have  been  shown  to  be  highly 
reliable,  especially  if  data  are  collected  from  30  incumbents  per  occupation.  These  descriptors 
have  also  been  shown  to  differentiate  between  occupations  in  sensible  ways.  Thus,  it  is  likely  that 
this  information  can  be  used  to  generate  occupational  clusters  highly  appropriate  for  the 
development  of  occupational  skill  standards. 

School-To-Work 

Efforts  in  this  area  focus  on  building  a school-to-work  transition  system  for  the  75 
percent  of  our  young  people  who  do  not  go  on  to  complete  a four-year  post-secondary  education. 
These  state-led  efforts  will  build  upon  existing  programs  to  prepare  students  to  meet  high 
academic  and  occupational  standards,  and  create  linkages  to  the  labor  market  system.  These 
efforts  require  the  skill  requirements  of  occupations  to  help  build  curricula  for  schools.  Again, 
the  0*NET  will  be  responsive  to  this  need,  with  the  skill  taxonomy  descriptors  providing  the 
relevant  data.  In  addition  to  demonstrating  appropriate  psychometric  properties,  data  collected 
using  the  skill  descriptors  demonstrates  meaningful  patterns  of  correlations  with  generalized 
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work  activities,  abilities  and  other  descriptors,  providing  support  for  the  construct  validity  of 
these  measures.  Accurate  data  concerning  the  skill  requirements  of  a broad  sampling  of 
occupations  will  provide  a good  foundation  for  efforts  to  prepare  students  for  the  workplace  and 
to  facilitate  a smooth  transition  from  school  to  work. 

Dislocated  Workers  and  One-Stops 

Work  in  this  area  seeks  to  consolidate  and  improve  an  array  of  existing  programs  targeted 
at  laid-off  workers.  Resources  and  incentives  have  been  provided  for  states  and  local 
communities  to  create  One-Stop  Career  Centers  where  employers  and  workers  will  have  easy 
access  to  job  and  training  information,  post-secondary  professional  and  technical  education,  adult 
basic  education,  job  matching,  and  counseling  services.  The  Centers  will  be  overseen  by  a 
workforce  investment  board.  The  One-Stop  Career  Centers,  proposed  as  part-of  the  Workforce 
Security  Act,  will  provide  laid-off  workers  with  job-search  assistance,  labor-market  information, 
and  training  to  enable  these  workers  to  take  control  of  their  careers,  allowing  them  to  move  into 
new  occupations.  In  this  case,  the  need  is  for  job  and  occupational  information  to  help  dislocated 
workers  find  new  jobs.  The  0*NET  can  help  with  part  of  this  requirement  by  providing  them 
with  detailed  information  about  skills,  abilities,  etc.,  needed  for  different  occupations.  Through 
either  formal  or  informal  means  these  workers  can  match  their  qualifications  to  the  occupational 
requirements. 

High-Performance  Workplaces 

The  0*NET  can  make  several  contributions  to  efforts  to  build  high-performance 
workplaces.  First,  the  content  model  provides  a comprehensive  description  of  high-performance 
business  practices,  which  has  a strong  foundation  in  research  and  theory,  as  well  as  demonstrated 
links  to  other  attempts  to  define  high-performance  or  high-involvement  workplaces.  Data 
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concerning  these  aspects  of  the  organizational  context  were  collected  from  both  organizational 
representatives  (e.g.,  personnel  managers)  and  incumbents  in  the  targeted  occupations. 
Organizational  representative  data  were  collected  from  over  600  organizations;  these  data 
provide  an  excellent  source  of  information  concerning  the  relationships  among  these  high- 
performance  practices,  their  relationships  with  other  frequently  studied  aspects  of  organizations, 
and  the  extent  to  which  these  practices  are  used  in  the  workplaces  sampled.  These  analyses 
revealed  that  business  practices  identified  as  high-performance  do  tend  to  covary  across 
organizations.  This  comprehensive  and  empirically-based  description  of  the  high-performance 
practices  used  in  workplaces  can  provide  a foundation  for  future  research  and  applications. 

In  addition,  high-performance  workplaces  have  been  described  as  using  data  to  make 
organizational  decisions  and  employing  state  of  the  art  business  practices.  0*NET’s  dynamic  and 
flexible  occupational  information  database  is  ideally  suited  to  support  these  activities. 

Although  certainly  not  exhaustive  of  important  workplace  initiatives,  we  believe  the 
0*NET  is  clearly  responsive  to  the  needs  of  these  kinds  of  programs.  Applications  related  to  the 
content  model  should  be  invaluable  for  helping  to  make  these  programs  successful.  While  more 
data  are  needed  before  this  potential  is  realized,  results  to  date  suggest  that  0*NET  will  benefit 
all  of  these  initiatives  substantially. 

Issues  in  Describing  Occupations 

There  are  many  ways  the  0*NET  might  be  used  to  describe  occupations.  For  example, 
profiles  of  means  in  tabular  or  graphic  form,  either  within  a particular  domain  or  across  domains 
would  provide  a comprehensive  picture  of  the  skill,  ability,  GW  A,  etc.  requirements  for  a job. 
For  many  applications,  however,  this  characterization  will  be  cumbersome,  with  too  much 
information  to  digest.  If  we  could  provide  a less  detailed  description,  for  example  showing  only 
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the  highest  level  or  most  important  requirements,  the  information  might  be  more  palatable  to 
some  users.  However,  there  are  several  available  options  for  selecting  these  descriptors:  (1)  we 
might  present  a certain  set  of  descriptors  from  a single  domain  or  a subset  of  the  domains;  (2)  we 
might  include  all  domains  and  simply  list  the  descriptors  with  the  highest  level  ratings  or  perhaps 
the  most  important  descriptors  from  each  domain;  and  (3.)  we  could  set  a cutoff  on  the  level  or 
importance  scales  and  list  all  descriptors  with  means  above  that  cutoff.  (This  last  approach  might 
result  in  no  descriptors  at  all  or  several  descriptors  from  each  domain,  for  a given  occupation.) 
Other  approaches,  perhaps  using  scores  aggregated  from  the  lowest  level,  could  also  be 
employed. 

Using  Raw  or  Standardized  Means 

One  important  issue,  regardless  of  which  approach  is  taken  is  whether  to  use  raw  mean 
ratings  within  each  occupation  or  to  standardize  the  mean  scores  for  each  descriptor  across  the 
occupations,  and  then  use  these  standardized  scores  to  rank  order  the  descriptors  within  an 
occupation.  The  descriptors  that  emerge  as  highest  level,  most  important,  etc.,  are  likely  to  be 
different  with  these  two  procedures.  This  is  because  there  are  substantial  descriptor  effects,  with 
some  descriptor  means  considerably  higher  than  others.  As  an  example,  the  Communicating 
With  Supervisors,  Peers,  or  Subordinates  GWA  domain  level  scale  has  a mean  of  4.14  across  the 
35  occupations,  whereas,  the  Specifying  Equipment  scale  mean  is  .87.  Thus,  the  Communicating 
descriptor  will  tend  to  be  high  in  the  rank  order  for  many  of  the  occupations.  This  is  appropriate 
for  applications  that  require  a depiction  of  the  highest  level,  most  important,  etc.,  descriptors  in 
an  absolute  sense.  For  other  applications,  standardized  scores  might  be  preferred.  That  is,  when 
we  want  to  obtain  a picture  of  the  highest  level  descriptors  relative  to  other  occupations,  for 
example,  using  the  standardized  scores  to  generate  the  rank  ordering  will  be  appropriate.  Note, 
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though,  that  the  standardized  score  method  is  very  sensitive  to  the  number  and  nature  of  the 
occupations  included  in  the  sample. 

\ 

To  provide  preliminary  information  concerning  the  relative  merits  of  using  raw  and 
standardized  scores  in  generating  occupation  descriptions,  we  computed  means  of  both  the  raw 
and  standard  scores  on  all  of  the  content  model  level  descriptors  for  four  occupations  with  a 
relatively  large  N and  selected  to  reflect  a wide  range  of  types  of  occupations:  (1)  General 
Managers  and  Top  Executives  (N  = 147);  (2)  Registered  Nurses  (N  = 97);  (3)  General  Office 
Clerks  (N  = 264);  and  (4)  Maintenance  Repairers,  General  Utility  (N  = 1 10).  We  used  the  level 
scales  rather  than  importance,  because  they  are  most  likely  to  be  the  retained  scale  should  either 
level  or  importance  be  eliminated.  Standard  scores  were  computed  by  adjusting  scores  for  each 
descriptor  to  have  a mean  of  zero  and  a standard  deviation  of  one  across  all  29  occupations 
included  in  these  analyses.  Accordingly,  at  this  point,  we  had,  for  each  content  model  domain, 
the  descriptors  rank  ordered  in  relation  to  level  means  by  raw  scores  and  by  standard  scores  for 
the  four  occupations. 

These  analyses  demonstrate  that  the  way  descriptors  are  rank  ordered  substantially  effects 
the  outcome  of  that  ranking.  One  aspect  of  these  findings  is  that  for  some  of  the  domains,  when 
raw  scores  are  used,  the  same  descriptor  appears  near  the  top  of  the  list  for  all  of  the  occupations. 
For  example,  with  GW  As,  Establishing  Relationships  is  ranked  first  for  the  Nurses  and  General 
Clerks,  third  for  the  Repairers,  and  fourth  for  the  Managers.  In  the  abilities  domain.  Managers 
and  Nurses  are  described  very  similarly,  with  Problem  Sensitivity,  Speech  Clarity,  Written 
Expression,  and  Oral  Expression  near  the  top  of  both  lists.  For  work  styles,  Dependability  is  top 
or  second  on  the  list  for  all  four  of  the  occupations.  Finally,  for  work  context,  Social  Interaction 
is  near  the  top  for  the  four  occupations.  This  phenomenon  is  not  evident  when  standard  scores 
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Table  13-2  (continued) 

Intercorrelations  of  Factor-Based  Composites  Across  Domains 
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Table  13-2  (continued) 

Intercorrelations  of  Factor-Based  Composites  Across  Domains 
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are  employed.  As  would  be  expected,  standardizing  has  the  effect  of  emphasizing  differences 
between  occupations. 

To  illustrate.  Table  15-2  summarizes  the  extent  to  which  the  ten  descriptors  with  the 
highest  mean  level  scale  ratings  overlap  across  pairs  of  occupations— that  is,  the  number  of  these 
“top  ten”  descriptors  that  are  identical.  Table  15-2  presents  this  information  for  both  raw  and 
standardized  descriptor  scores.  As  mentioned  above,  for  GW  As  there  is  a great  deal  of  overlap  in 
the  occupation  descriptions  that  would  be  generated  for  these  four  occupations  using  the  raw 
descriptor  data.  Between  three  and  six  of  the  top  ten  descriptors  overlap  across  the  various  pairs 
of  occupations.  The  overlap  is  much  less  extensive  for  the  standardized  data,  ranging  from  zero 
to  three  across  pairs  of  occupations.  Results  are  similar  for  skills,  abilities,  knowledges,  and  work 
context.  For  the  work  style  and  occupational  value  domains  the  differences  across  the  raw  versus 
standardized  data  are  much  smaller.  However,  these  latter  two  domains  contain  fewer  descriptors 
(e.g.,  there  are  only  17  work  styles  descriptors),  so  considerable  overlap  in  the  top  ten  would  be 
expected  simply  by  chance. 

Overall,  we  can  see  that  using  the  standardized  scores  in  the  job  descriptions  makes  these 
four  occupations  look  substantially  more  unique  than  when  raw  scores  are  used.  This  is  desirable, 
in  that  the  four  occupations  were  selected  to  represent  a wide  range  of  job  content.  Accordingly, 
the  standardized  scores  appear  to  be  most  appropriate  for  this  type  of  job  description  application. 

Using  a Fixed  Number  of  Descriptors  or  a Cutoff  Score 

Table  15-3  provides  data  relevant  to  decisions  concerning  whether  a fixed  number  of 
descriptors  should  be  used  in  each  occupation  description  or  a particular  cutoff  score  should  be 
used  to  identify  the  descriptors  to  be  included.  Recall  that  there  are  two  general  approaches  that 
could  be  taken  here.  One  is  to  simply  list  a certain  number  of  descriptors  for  each  domain  that 
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have  the  highest  level  ratings  (e.g.,  the  top  five  or  the  top  ten).  The  second  is  to  select  a cutoff 
score  on  the  level  scale  and  then  list  all  descriptors  with  means  above  that  cutoff.  The  latter 
approach  has  some  appeal  because  it  allows  for  control  over  the  average  level  of  descriptors  that 
would  be  included  in  occupation  descriptions.  Table  15-3  shows  how  many  descriptors  would  be 
selected  for  describing  each  of  the  four  example  occupations  if  a cutoff  score  of  five  on  the 
seven-point  level  scale  was  used,  and  the  number  that  would  be  selected  if  a cutoff  score  of  four 
was  used.  This  table  shows  that  there  are  systematic  differences  in  the  numbers  of  descriptors 
that  would  be  included  in  the  resulting  descriptions  across  occupations.  A parallel  set  of  analyses 
was  conducted  using  the  standardized  descriptor  scores.  Results  showed  a very  similar  pattern  to 
those  for  the  raw  data  and  are  not  presented  here.  Table  15-3  also  shows  that  the  numbers  of 
descriptors  chosen  using  these  absolute  cutoffs  varies  systematically  across  domains.  If  this 
approach  is  taken,  it  is  likely  that  different  cutoff  scores  will  need  to  be  used  for  different 
domains. 

In  order  to  illustrate  the  extent  to  which  choosing  a set  number  of  descriptors  (e.g.,  ten) 
would  lead  to  occupation  descriptions  based  on  descriptors  with  systematically  different  mean 
level  ratings.  Table  15-4  presents  the  mean  level  rating  of  the  tenth  descriptor  (i.e.,  the  descriptor 
with  the  tenth  highest  mean  level  rating)  for  each  domain  for  each  of  these  four  occupations.  The 
differences  in  these  level  means  across  occupations  are  generally  quite  large.  For  example,  the 
tenth  highest  skills  level  rating  for  Managers  and  Executives  is  5.2,  while  for  clerks  the  tenth 
highest  descriptor  was  rated  only  2.8.  Again,  these  results  vary  by  domain.  For  example,  for  work 
styles  the  average  level  of  the  tenth  highest  descriptor  is  much  more  similar  across  occupations 
than  for  skills. 
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The  analyses  presented  to  this  point  have  focused  on  only  four  occupations.  To  obtain 
some  preliminary  information  concerning  the  extent  to  which  these  results  are  likely  to 
generalize  to  other  occupations,  we  conducted  this  same  set  of  analyses,  for  the  GWA  domain 
only,  for  all  29  occupations  for  which  data  are  currently  available.  The  mean  number  of  GWA 
descriptors  that  would  be  selected  using  a cutoff  of  five  or  greater  on  the  level  scale  across  all  29 
occupations  is  2.45,  and  the  number  per  occupation  ranges  from  0 to  20.  For  a cutoff  of  four  or 
greater  on  the  level  scale  the  mean  number  selected  across  occupations  would  be  7.24,  and  the 
number  per  occupation  ranges  from  0 to  28.  The  mean  level  score  of  the  tenth  GWA  descriptor 
(i.e.,  the  GWA  descriptor  with  the  tenth  highest  level  scale  mean)  across  all  29  occupations  is 
3.32,  and  the  means  for  individual  occupations  range  from  5.5  (for  Education  Administrators)  to 
1.8  (for  Truck  Drivers).  Thus,  the  results  for  the  four  selected  occupations  appear  to  be 
reasonably  representative  of  those  for  the  29  occupations  currently  available. 

Using  Single  or  Multiple  Domains 

The  discussion  so  far  has  focused  on  selecting  descriptors  from  within  a single  domain  to 
describe  occupations.  Another  important  issue  is  the  extent  to  which  considering  more  than  one 
domain  simultaneously  will  affect  the  results,  and  whether  this  approach  would  provide  more  or 
less  appropriate  data  for  certain  applications.  It  could  be  argued  that  important  information  is 
overlooked  when  data  from  the  various  content  domains  are  viewed  independently.  For  example, 
if  we  choose  the  top  five  descriptors  from  each  content  domain  to  describe  each  occupation, 
differences  in  the  relative  importance  of  the  various  content  domains  across  occupations  would 
not  be  captured.  It  is  possible  that  abilities,  in  general,  are  more  appropriate  for  describing  the 
highest  level  or  most  important  requirements  for  some  occupations,  whereas  for  other 
occupations  skills  are  generally  more  appropriate.  For  example,  one  theory  would  be  that  for 
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higher  level  occupations,  such  as  Managers  and  Executives,  skills  will  be  relatively  more 
important  than  abilities. 

There  are  two  ways  this  issue  can  be  addressed.  First,  we  could  simply  make  an  overall 
assessment  of  the  importance  or’ appropriateness  of  descriptors  from  each  domain  for  describing 
a particular  occupation.  For  example,  the  level  scores  for  descriptors  within  each  domain  could 
be  averaged  within  each  occupation.  This  would  provide  an  overall  indication  of  the  level  of 
descriptor  scores  for  a particular  domain  for  each  occupation.  These  could  even  be  used  to 
generate  percentile  ratings  that  essentially  indicate  the  importance  of  that  domain  for  a given 
occupation,  relative  to  all  other  occupations.  Another  way  to  examine  this  would  be  to  count  the 
number  or  percentage  of  descriptors  in  a particular  domain  for  a particular  occupation  above  a 
certain  level.  These  overall  assessments  of  the  importance  of  each  domain  could  then  be 
compared  with  those  for  other  domains. 

Another  way  to  address  this  issue  is  to  rank  order  descriptors  from  two  or  more  domains 
simultaneously  and  then  determine  the  extent  to  which  descriptors  at  the  top  of  the  list  tend  to 
represent  different  domains  for  different  occupations.  Again  this  could  be  done  using  either  raw 
or  standardized  descriptor  information.  To  the  extent  that  there  are  overall  differences  in  the 
average  level  ratings  across  domains,  this  will  be  reflected  in  the  joint  rankings,  so  standardized 
data  are  arguably  more  appropriate  for  this  purpose.  In  order  to  provide  some  preliminary 
information  concerning  how  occupation  descriptions  might  differ  if  we  consider  more  than  one 
domain  simultaneously,  we  chose  three  pairs  of  domains-skills  and  abilities;  skills  and 
knowledges;  and  work  styles  and  abilities~and  rank  ordered  the  standardized  descriptors 
simultaneously  for  these  pairs  within  each  of  the  four  occupations.  Table  15-5  summarizes  the 
results.  These  findings  suggest  that  there  are  systematic  differences  across  occupations  in  the 
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importance  of  the  various  content  domains  for  describing  occupation  requirements.  For  example, 
of  the  top  ten  skill  and  ability  descriptors  for  Managers  and  Executives,'  all  ten  are  skills,  whereas 
for  General  Clerks  all  of  the  top  ten  are  abilities.  The  other  two  occupations  have  a mix  of  skills 
and  abilities  in  the  top  ten. 

Summary 

In  general,  the  analyses  described  here  show  that  the  0*NET  descriptors  can  be  used  to 
generate  sensible  occupation  descriptions.  Each  of  the  approaches  to  generating  these  occupation 
descriptions  has  merit,  but  no  single  approach  emerged  as  superior.  Standardized  data  generally 
led  to  more  interesting  and  conceptually  appealing  occupation  descriptions,  but  raw  data  are  still 
likely  to  be  most  appropriate  for  at  least  some  purposes.  For  example,  educators  may  be  more 
interested  in  the  descriptors  rated  to  have  the  highest  level  requirements  in  an  absolute  sense, 
because  issues  in  this  area  are  not  generally  related  to  differentiating  between  occupations  but 
rather  identifying  the  aspects  of  educational  curricula  that  could  enhance  performance  in  the 
workplace.  For  these  applications,  the  finding  that  some  activities  are  required  at  a relatively  high 
level  across  a wide  variety  of  occupations  could  be  viewed  as  highlighting  those  activities  and 
related  worker  requirements  that  are  important  to  include  in  educational  curricula  regardless  of 
students’  occupational  plans.  Applications  related  to  placement  or  career  search  decisions  are 
likely  to  benefit  most  from  information  concerning  the  relative  descriptor  levels  across 
occupations,  that  is,  standardized  data.  Comparisons  of  occupation  descriptions  generated  using  a 
level  scale  cutoff  score  versus  an  absolute  number  of  descriptors  per  domain  did  not  reveal  that 
either  method  was  clearly  superior.  A combination  of  these  two  approaches  is  likely  to  be  best 
for  many  purposes,  and  an  example  of  how  these  approaches  might  be  combined  is  presented 
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next.  In  addition,  there  are  large  differences  across  domains,  and  it  is  likely  that  decision  rules 
used  to  generate  occupation  descriptions  will  necessarily  vary  by  domain. 

Using  0*NET  to  Generate  Occupation  Descriptions:  An  Example  Application 

Any  application  of  the  0*NET  data  must  begin  by  clearly  articulating  the  purpose  of  the 
application  and  the  needed  information,  and  identifying  the  content  domain  or  domains  that  are 
relevant  for  this  purpose.  The  0*NET  contains  over  300  descriptors,  and  focusing  on  only  the 
most  relevant  aspects  of  the  content  domain  will  be  more  efficient  and  effective  for  many 
applications.  In  addition,  even  within  a particular  content  domain  only  a subset  of  variables  may 
be  relevant  for  some  purposes.  For  example,  each  GWA  descriptor  has  both  frequency  and  level 
data,  and  the  training,  education,  licensure,  and  experience  domain  contains  a variety  of  different 
types  of  information.  Finally,  our  previous  discussion  of  issues  in  generating  occupation 
descriptions  suggests  a variety  of  other  issues  that  may  need  to  be  considered  for  a particular 
application.  Most  importantly,  users  must  determine  whether  absolute  or  relative  occupation 
description  information  is  most  appropriate  for  a given  purpose. 

We  now  demonstrate  what  is  likely  to  be  a commonly  used  application— generating 
written  occupation  descriptions  for  internal  human  resources  users.  For  purposes  of  illustration, 
we  will  employ  a particular  set  of  decision  rules  to  generate  these  descriptions:  The  decision  rules 
actually  used  in  practice  will  likely  vary  according  to  the  specific  application.  For  this  illustration 
we:  (1)  use  standard  scores  rather  than  raw  scores  to  emphasize  the  differences  between 
occupations;  (2)  have  at  least  one  descriptor  per  domain  represented  in  the  content  model;  (3) 
include  three  or  four  descriptors  if  they  have  very  close  to  the  same  mean  standard  scores  (4) 
attempt  to  minimize,  however,  the  number  of  descriptors  included  (a  very  long  occupation 
description  may  not  be  desirable);  (5)  include'the  operational  definitions  of  the  descriptors  to 
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provide  a richer  description  of  the  occupation  requirements;  and  (6)  include  the  four  most 
frequently  performed  occupation-specific  tasks.  Figure  15-1  shows  how  the  occupation 
descriptions  might  actually  look  for  four  example  occupations. 

Conclusions 

The  potential  applications  of  0*NET  are  many.  The  contributions  of  this  database  to 
personnel  selection,  job  seeking,  learning  about  jobs  and  occupations,  targeted  training  program 
development',  certification  of  competency  and  mastery,  and  research  on  occupations  and  the 
workplace  can  be  tremendous. 

Preliminary  data  on  between  30  and  37  occupations  are  promising  with  respect  to  0*NET 
applications.  Reliability  results  are  especially  encouraging.  Findings  to  date  demonstrate  that  job 
incumbents  can  provide  reliable  information  about  their  occupations.  This  in  turn  means  that  the 
database  can  be  used  with  confidence.  The  discriminant  function  analysis  results  are  also 
encouraging.  The  content  model  descriptors  accomplish  what  they  were  intended  to 
accomplish — they  differentiate  between  occupations.  These  two  sets  of  results,  in  particular, 
indicate  that  the  applications  we  foresee,  and  probably  those  that  will  emerge  later,  should  be 
successfully  accomplished. 

A variety  of  issues  related  to  using  the  0*NET  to  describe  occupations  were  discussed  in 
this  chapter,  and  preliminary  analyses  exploring  the  merits  of  various  approaches  to  occupation 
description  are  presented.  Additional  analyses  are  needed  to  clarify  some  of  these  issues,  but  the 
purpose  of  any  given  application  will  always  be  a critical  factor  in  decisions  concerning  which 
approach  is  best.  Finally,  we  were  able  to  demonstrate  how  the  0*NET  database  might  be  used 
to  generate  written  occupation  descriptions  for  use  in  human  resource  management.  It  is  our 
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belief  that  0*NET  should  be  an  enormously  important  national  resource  for  dealing  with 
problems  related  to  occupations,  the  workplace,  and  productivity. 
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Figure  15-1 

Example  Job  Descriptions  for  Four  Occupations 


Job  Description:  General  Manager  & Top  Executive 


General  Description 


Top  and  middle  managers  whose  duties  and  responsibilities  are  too  diverse  and  general  in  nature  to  be 
classified  in  any  functional  or  line  area  of  management  and  administration.  These  managers  generally 
work  through  departmental  or  subordinate  executives. 


Critical  Tasks 


Consults  with  staff  to  assist  in  making  decisions  and  coordinating  activities. 

Directs  and  coordinates  activities  of  workers  to  ensure  continuing  operations,  maximize  returns  on 
investments,  and  increase  productivity. 

Interprets,  and  explains  policies,  rules,  regulations,  and  laws  to  organizations  and  individuals. 
Establishes  and  maintains  a record  keeping  system  for  the  organization. 


Work  Activity  Requirements 


Monitoring  and  Controlling  Resources  - Monitoring  and  controlling  resources  and  overseeing  the 
spending  of  money. 

Guiding,  Directing,  and  Motivating  Subordinates  - Providing  guidance  and  direction  to  subordinates, 
including  setting  performance  standards  and  monitoring  subordinates. 

Staffing  Organizational  Units  - Recruiting,  interviewing,  selecting,  hiring,  and  promoting  persons  in  the 
organization. 


Work  Context  Description 


Private  work  area. 

Setting  where  you  have  influence. 

Setting  where  the  impact  of  your  decisions  is  high. 
Setting  where  you  have  high  decision  latitude. 
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Figure  15-1  (continued) 

Example  Job  Descriptions  for  Four  Occupations 

Job  Description:  General  Manager  & Top  Executive  (continued) 


1 Knowledge  Requirements 

Administration  and  Management  - Knowledge  of  principles  and  processes  involved  in  business  and 
organization  planning,  coordination,  and  execution. 

Personnel  and  Human  Resources  - Knowledge  of  policies  and  practices  involved  in  personnel/human 
resource  functions. 

Economics  and  Accounting  - Knowledge  of  economic  and  accounting  principles  and  practices,  the 
financial  markets,  banking,  and  the  analysis  and  reporting  of  financial  data. 


| Skill  Requirements 

Management  of  Financial  Resources  - Determining  how  money  will  be  spent  to  get  the  work  done,  and 
accounting  for  these  expenditures. 

Systems  Evaluation  - Looking  at  many  indicators  of  system  performance,  taking  into  account  their 
accuracy. 

Identification  of  Downstream  Consequences  - Determining  the  long-term  outcomes  of  change  in 
operations. 

Management  of  Personnel  Resources  - Motivating,  developing,  and  directing  people  as  they  work , 
identifying  the  best  people  for  the  job. 


Ability  Requirements 


Fluency  of  Ideas  - The  ability  to  come  up  with  a number  of  ideas  about  a given  topic. 

Originality  - The  ability  to  come  up  with  unusual  or  clever  ideas  about  a given  topic  or  situation,  or  to 
develop  creative  ways  to  solve  a problem. 

Inductive  Reasoning  - The  ability  to  combine  separate  pieces  of  information,  or  specific  answers  to 
problems,  to  form  general  rules  or  conclusions. 


1333 


Figure  15-1  (continued) 

Example  Job  Descriptions  for  Four  Occupations 


Job  Description:  General  Manager  & Top  Executive  (continued) 
Work  Style  Requirements 

Leadership  Orientation  - Job  requires  a willingness  to  lead,  take  charge  and  offer  opinions  and  direction. 

Analytical  Thinking  - Job  requires  analyzing  information,  and  using  logic  to  address  work  or  job  issues 
and  problems. 

Initiative  - Job  requires  being  willing  to  take  on  responsibilities  and  challenges. 

Achievement/Effort  - Job  requires  establishing  and  maintaining  personally  challenging  achievement 
goals,  and  exerting  effort  toward  task  mastery. 

Occupational  Value  Requirements 

Authority  - Provide  direction  and  instruction  to  others. 

Social  Status  - Are  looked  up  to  by  others  in  organization  and  community. 

Creativity  - Get  to  try  out  own  ideas. 

Compensation  - Are  well  paid  in  comparison  to  others. 


Figure  15-1  (continued) 

Example  Job  Descriptions  for  Four  Occupations 


Job  Description:  Registered  Nurse 

General  Description 

Administer  nursing  care  to  ill  or  injured  persons.  Licensing  or  registration  required.  Include 
administrative,  public  health,  industrial,  private  duty,  and  surgical  nurses. 

Critical  Tasks 

Observes,  evaluates,  and  records  patient  data. 

Administers  injections,  medications,  and  treatments. 

Consults  with  other  medical  professionals  on  policy  and  patient  care. 

Counsels  and  provides  support  for  patients  and  families. 

Work  Activity  Requirements 

Assisting  and  Caring  for  Others  - Providing  assistance  or  personal  care  to  others. 

Teaching  Others  - Identifying  educational  needs  and  developing  (or  having  others  develop)  formal 
training  programs  or  classes. 

Documenting/Recording  Information  - Entering,  transcribing,  recording,  storing,  or  maintaining 
information  in  either  written  form  or  by  electronic/magnetic  recording. 

Work  Context  Description 

Works  around  diseases/infections. 

Consequences  of  errors  are  great. 

Special  uniform  is  required. 


Figure  15-1  (continued) 

Example  Job  Descriptions  for  Four  Occupations 

Job  Description:  Registered  Nurse  (continued) 


Knowledge  Requirements 


Medicine  and  Dentistry  - Knowledge  of  the  information  and  techniques  needed  to  diagnose  and  treat 
injuries,  diseases,  and  deformities. 

Biology  - Knowledge  of  plant  and  animal  living  tissue,  cells,  organisms  and  entities,  including  their 
functions,  interdependencies,  and  interactions  with  each  other  and  the  environment. 

Therapy  and  Counseling  - Knowledge  of  information  and  techniques  needed  to  rehabilitate  physical  and 
mental  ailments,  and  to  provide  career  guidance  including  alternative  treatments,  rehabilitation 
equipment  and  its  proper  use,  and  methods  to  evaluate  treatment  effects. 


Skill  Requirements 


Science  - Using  scientific  methods  to  solve  problems. 

Social  Perceptiveness  - Being  aware  of  others’  reactions  and  understanding  why  they  react  the  way  they 
do. 

Operation  Monitoring  - Watching  gauges,  dials,  or  other  indicators  to  make  sure  a machine  is  working 
properly. 

Ability  Requirements 


Problem  Sensitivity  - The  ability  to  tell  when  something  is  wrong  or  is  likely  to  go  wrong. 

Inductive  Reasoning  - The  ability  to  combine  separate  pieces  of  information,  or  specific  answers  to 
problems,  to  form  general  rules  or  conclusions. 

Arm-Hand  Steadiness  - The  ability  to  keep  the  hand  and  arm  steady  while  making  an  arm  movement  or 
while  holding  the  arm  and  hand  in  one  position. 

Speech  Recognition  - The  ability  to  identify  and  understand  the  speech  of  another  person. 
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Figure  15-1  (continued) 

Example  Job  Descriptions  for  Four  Occupations 

Job  Description:  Registered  Nurse  (continued) 


Work  Style  Requirements 


Concern  for  Others  - Job  requires  being  sensitive  to  others’  needs  and  feelings,  and  being  understanding 
and  helpful  to  others  on  the  job. 

Adaptability/Flexibility  - Job  requires  being  open  to  change  (positive  or  negative)  and  to  considerable 
variety  in  the  workplace. 

Self-Control  - Job  requires  maintaining  composure,  keeping  emotions  in  check,  controlling  anger,  and 
avoiding  aggressive  behavior  even  in  very  difficult  situations. 

Social  Orientation  - Job  requires  preferring  to  work  with  others  rather  than  alone  and  being  personally 
connected  with  others  on  the  job. 

Persistence  - Job  requires  persistence  in  the  face  of  obstacles  on  the  job. 


Occupational  Value  Requirements 


Social  Service  - Get  to  do  things  for  other  people. 

Security  - Have  steady  employment. 

Social  Status  - Are  looked  up  to  by  others  in  the  organization  and  community. 
Achievement  - Get  a feeling  of  accomplishment  on  the  job. 
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Figure  15-1  (continued) 

Example  Job  Descriptions  for  Four  Occupations 


Job  Description:  General  Office  Clerk 


General  Description 


Perform  duties  too  varied  and  diverse  to  be  classified  in  any  specific  office  clerical  occupation.  Clerical 
duties  may  be  assigned  in  accordance  with  the  office  procedures  of  individual  establishments  and  may 
include  a combination  of  bookkeeping,  typing,  stenography,  office  machine  operation,  and  filing. 


Critical  Tasks 


Operates  office  machines,  such  as  photocopier,  fax,  and  personal  computer. 

Answers  telephone,  responds  to  requests,  delivers  messages,  and  runs  errands. 

Communicates  with  customers,  employees,  and  other  individuals  to  disseminate  or  explain  information. 
Compiles,  copies,  sorts,  and  files  records  of  office  activities,  business  transactions,  and  other  activities. 


Work  Activity  Requirements 


Interacting  with  Computers  - Controlling  computer  functions  by  using  programs,  setting  up  functions, 
writing  software,  or  otherwise  communicating  with  computer  systems. 

Processing  Information  - Compiling,  coding,  categorizing,  calculating,  tabulating,  auditing,  verifying,  or 
processing  information  or  data. 


Documenting/Recording  Information  - Entering,  transcribing,  recording,  storing,  or  maintaining 
information  in  either  written  form  or  by  electronic/magnetic  recording. 


Work  Context  Description 


Work  station  requires  sitting. 

Uses  computer  and  telephone  to  communicate. 

Knowledge  Requirements 


Clerical  - Knowledge  of  administrative  and  clerical  procedures  and  systems  such  as  word  processing 
systems,  filing  and  records  management  systems,  stenography  and  transcription,  forms  design  principles, 
and  other  office  procedures  and  terminology. 


Skill  Requirements 


Operation  and  Control  - Controlling  operations  of  equipment  or  systems. 
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Figure  15-1  (continued) 

Example  Job  Descriptions  for  Four  Occupations 

Job  Description:  General  Office  Clerk  (continued) 


Ability  Requirements 

Wrist-Finger  Speed  - The  ability  to  make  fast,  simple,  repeated  movements  of  the  fingers,  hands,  and 
wrists. 


| Work  Style  Requirements 

Attention  to  Detail  - Job  requires  being  careful  about  detail  and  thorough  in  completing  work  tasks. 

Occupational  Value  Requirements 

Activity  - Are  busy  all  the  time. 
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Figure  15-1  (continued) 

Example  Job  Descriptions  for  Four  Occupations 

Job  Description:  Maintenance  Repairer,  General  Utility 


General  Description 


Perform  work  involving  two  or  more  maintenance  skills  to  keep  machines,  mechanical  equipment,  or 
structure  of  an  establishment  in  repair.  Duties  may  involve  pipefitting,  boilermaking,  insulating,  welding, 
machining,  machine  and  equipment  repairing,  carpentry,  and  electrical  work.  May  also  include  planning 
and  laying  out  of  work  relating  to  repairs;  repairing  electrical  and/or  mechanical  equipment;  installing, 
aligning,  and  balancing  new  equipment;  and  repairing  buildings,  floors,  or  stairs.  These  workers  are 
generally  found  in  small  establishments  where  specialization  in  maintenance  work  is  impractical. 


Critical  Tasks 


Installs  new  or  repaired  parts. 

Inspects  and  tests  machinery  and  equipment  to  diagnose  machine  malfunctions. 
Cleans  and  lubricates  shafts,  bearings,  gears,  and  other  parts  of  machinery. 
Dismantles  and  reassembles  defective  machines  and  equipment. 

Work  Activity  Requirements 


Repairing  and  Maintaining  Mechanical  Equipment  - Fixing,  servicing,  aligning,  setting  up,  adjusting, 
and  testing  machines,  devices,  moving  parts,  and  equipment  that  operate  primarily  on  the  basis  of 
mechanical  (not  electronic)  principles. 


Drafting,  Laying  Out,  and  Specifying  Technical  Devices,  Parts,  and  Equipment  - Providing 
documentation,  detailed  instruction,  drawings,  or  specifications  to  inform  others  about  how  devices, 
parts,  equipment,  or  structures  are  to  be  fabricated,  constructed,  assembled,  modified,  maintained,  or 
used.  - 


Operating  Vehicles,  Mechanized  Devices,  or  Equipment  - Running,  maneuvering,  navigating,  or  driving 
vehicles  or  mechanized  equipment,  such  as  forklifts,  passenger  vehicles,  aircraft,  or  water  craft. 

Inspecting  Equipment,  Structures,  or  Materials  - Inspecting  or  diagnosing  equipment,  structures  or 
materials  to  identify  the  causes  of  errors  or  other  problems  or  defects. 


Work  Context  Description 


Climbs  ladders,  scaffolds,  poles,  etc. 


Works  in  high  places. 

Wears  production  or  maintenance  worker  clothes. 
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Figure  15-1  (continued) 

Example  Job  Descriptions  for  Four  Occupations 

Job  Description:  Maintenance  Repairer,  General  Utility  (continued) 


Knowledge  Requirements 


Mechanical  - Knowledge  of  machines  and  tools,  including  their  designs,  uses,  benefits,  repair,  and 
maintenance. 

Building  and  Construction  - Knowledge  of  materials,  methods,  and  the  appropriate  tools  to  construct 
objects,  structures,  and  buildings. 


Skill  Requirements 


Repairing  - Repairing  machines  or  systems  using  the  needed  tools. 

Installation  - Installing  equipment,  machines,  wiring,  or  programs  to  meet  specifications. 

Equipment  Maintenance  - Performing  routine  maintenance  and  determining  when  and  what  kind  of 
maintenance  is  needed. 

Troubleshooting  - Determining  what  is  causing  an  operating  error  and  deciding  what  to  do  about  it. 

Operation  Monitoring  - Watching  gauges,  dials,  or  other  indicators  to  make  sure  a machine  is  working 
properly. 

Ability  Requirements 


Static  Strength  - The  ability  to  exert  maximum  muscle  force  to  lift,  push,  pull,  or  carry  objects. 

Control  Precision  - The  ability  to  quickly  and  repeatedly  make  precise  adjustments  in  moving  the 
controls  of  a machine  or  vehicle  to  exact  positions. 

Gross  Body  Equilibrium  - The  ability  to  keep  or  regain  one’s  body  balance  or  stay  upright  when  in  an 
unstable  position. 


Work  Style  Requirements 


Analytical  Thinking  - Job  requires  analyzing  information,  and  using  logic  to  address  work  or  job  issues 
and  problems. 

Innovation  - Job  requires  creativity  and  alternative  thinking  to  come  up  with  new  ideas  for  and  answers 
to  work-related  problems. 


Independence  - Job  requires  developing  one’s  own  ways  of  doing  things,  guiding  oneself  with  little  or  no 
supervision,  and  depending  on  oneself  to  get  things  done. 


1341 


Figure  15-1  (continued) 

Example  Job  Descriptions  for  Four  Occupations 

Job  Description:  Maintenance  Repairer,  General  Utility  (continued) 


Occupational  Value  Requirements 


Variety  - Have  something  different  to  do  every  day. 
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Table  15-1 

Comparison  of  Correlations  Within  and  Between  Second  Order  Categories  bv  Content  Domain 


Content  Domain 

Descriptors  From  the 
Same  Category 

Descriptors  From 
Different  Categories 

Mean 

Minimum 

Maximum 

Mean  Minimum  Maximum 

Generalized  Work  Activities 

.61 

-.26 

.88 

.43 

-.65 

.95 

Knowledges 

.58 

-.07 

.88 

.40 

-.23 

.95 

Skills 

.67 

.39 

.74 

.64 

-.21 

.97 

Abilities 

.79 

.13 

.95 

.26 

-.41 

.88 

Work  Styles 

.69 

.27 

.83 

.42 

-.47 

.84 

Occupational  Values 

.27 

-.26 

.88 

.24 

-.65 

.95 

Note.  N = 29. 


Table  15-2 

Overlap  in  Top  Ten  Descriptors  Across  Occupations  for  Raw  and  Standardized  Scores 


Mgr  ./Exec. 

Content  Domain & Nurse 

Generalized  Work  Activities: 

Raw  Scores  3 

Standardized  Scores  3 

Knowledges: 

Raw  Scores  . 3 

Standardized  Scores  3 

Skills: 

Raw  Scores  4 

Standardized  Scores  1 

Abilities: 

Raw  Scores  8 

Standardized  Scores  4 

Work  Styles: 

Raw  Scores  8 

Standardized  Scores  7 

_ Occupational  Values: 

Raw  Scores  9 

Standardized  Scores  6 

Work  Context: 

Raw  Scores  5 

Standardized  Scores  1 


Occupations  Compared 

Mgr  ./Exec.  Mgr  ./Exec.  Nurse  & Nurse  & Clerk  & 

& Clerk  & Mnt.  Rep.  Clerk  Mnt.  .Rep  Mnt.  Rep. 


4 5 

0 0 


5 3 6 

1 0 1 


9 3 

2 0 


4 4 3 

1 1 1 


4 

0 


3 

1 


9 2 2 

2 1 3 


6 

1 


4 8 5 4 

0 10  0 


7 

5 


7 9 7 8 

6 5 4 5 


8 8 9 

2 6 4 


9 8 

4 5 


5 4 6 7 6 

1 0 0 0 0 


Note.  Occupation  titles  are  abbreviated:  Mgr./Exec.  = General  Managers  & Top  Executives;  Nurses 
= Registered  Nurses;  Clerks  = General  Office  Clerks;  Mnt.  Rep.  = Maintenance  Repairers,  General 
Utility.  Numbers  in  this  table  represent  the  extent  to  which  the  ten  descriptors  with  the  highest  level 
ratings  overlap  across  pairs  of  occupations— that  is,  the  number  of  these  “top  ten”  descriptors  that  are 
identical.  For  example,  the  “3”  in  the  upper  left-hand  comer  indicates  that  Mgr  ./Exec,  and  Nurse 
have  three  of  their  top  ten  Generalized  Work  Activities  in  common  using  raw  scores. 
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Table  15-3 

Number  of  Descriptors  Included  in  Job  Descriptions  Based  on  Two  Level  Scale  Decision  Rules 


Numbers  of  Descriptors  Included 


Content  Domain  and 

Managers/ 

General 

Maintenance 

Decision  Rule 

Executives 

Nurses 

Clerks 

Repairers 

Generalized  Work 
Activities: 


5 or  above 

5 

3 

0 

0 

4 or  above 

22 

10 

3 

7 

Knowledges: 
5 or  above 

0 

1 

1 

0 

4 or  above 

6 

4 

1 

1 

Skills: 

5 and  Above 

19 

6 

0 

3 

4 and  Above 

33 

18 

1 

9 

Abilities: 

5 and  Above 

7 

9 

0 

0 

4 and  Above 

18 

19 

5 

18 

Work  Styles: 

5 or  above 

17 

17 

9 

9 

4 or  above 

17 

17 

17 

17 

Occupational  Values:3 

4.5  or  above 

2 

1 

0 

0 

4 or  above 

8 

6 

5 

3 

Occupational  Values  are  rated  on  a 1 to  5 scale. 

Table  15-4 


Mean  Level  Scale  Ratines  for  theTenth  Highest  Descriptor  in  Each  Domain  for  Each  Occupation 


Content  Domain 

Managers/ 

Executives 

Nurses 

General 

Clerks 

Maintenance 

Repairers 

Generalized  Work  Activities 

' 4.5 

4.0 

3.0 

3.8 

Knowledges 

3.2 

3.3 

1.5 

2.1 

Skills 

5.2 

4.7 

2.8 

3.9 

Abilities 

4.8 

4.8 

3.5 

4.2 

Work  Styles 

5.6 

5.9 

4.9 

4.9 

Occupational  Values8 

4.0 

3.8 

3.6 

3.7 

8Occupational  Values  are  rated  on  a 1 to  5 scale. 
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Table  15-5 

Composition  of  Top  Ten  Descriptors  for  Pairs  of  Domains  Rank  Ordered  Simultaneously 


Occupation 


Domain  Pair 

Managers/ 

Executives 

Nurses 

General 

Clerks 

Maintenance 

Repairers 

Skills  and 
Abilities 

10  skills 

6 skills 
4 abilities 

10  abilities 

7 skills 
4 abilities 

Skills  and 
Knowledges 

8 skills 
2 knowledges 

3 skills 
7 knowledges 

3 skills 
7 knowledges 

7 skills 
3 knowledges 

Abilities  and 
Work  Styles 

8 abilities 
2 work  styles 

9 abilities 
1 work  style 

7 abilities 
3 work  styles 

10  abilities 

Note.  Based  on  standardized  descriptor  data. 
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Chapter  16 

Occupation-Specific  Descriptors: 

Approaches,  Procedures,  and  Findings 

Michael  D.  Mumford 
Christopher  E.  Sager 
Wayne  A.  Baughman 
Ruth  A.  Childs 

American  Institutes  for  Research 


The  intent  of  this  chapter  is  to  describe  a set  of  procedures  that  might  be  used  to  gather 
occupation-specific  information  and  organize  this  kind  of  descriptive  data  in  terms  of  the  broader 
cross-occupation  variables  described  in  Chapters  3 through  1 1.  In  this  chapter,  we  present  the 
results  obtained  in  a series  of  pilot  studies  intended  to  assess  the  feasibility  of  applying  the 
suggested  procedures.  First,  though,  we  consider  what  is  perhaps  the  most  commonly  used  kind 
of  occupation-specific  descriptor— job  tasks  (McCormick,  1979a). 

Virtually  all  of  the  foregoing  chapters  have  focused  on  analyses  of  measures  that  might  be 
used  to  describe  most  occupations.  This  kind  of  cross-occupation  descriptive  system  provides  the 
foundation  for  a system  intended  to  answer  questions  about  a number  of  occupations.  These 
cross-occupation  descriptors,  however,  do  not  and,  in  fact,  cannot  address  all  of  the  various  types 
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of  descriptors  used  to  describe  people’s  jobs.  More  specifically,  these  taxonomies  of  cross- 
occupation  descriptors  do  not  provide  occupation-specific  information— such  as  the  tasks  and 
occupation-specific  skills— that  only  apply  to  a single  occupation  or  to  a narrowly  defined  job 
family. 

Many  questions  can  be  answered  without  referring  to  occupation-specific  information 
(Pearlman,  1993).  But  as  McCage  (1994)  points  out,  occupation-specific  information  may  be 
required  to  answer  other  important  questions.  For  example,  occupation-specific  information  may 
be  necessary  to  specify  training  requirements,  develop  position  descriptions,  and  undertake  the 
redesign  of  jobs. 

The  various  applications  of  occupation-specific  information  clearly  argue  for  inclusion  of 
certain  types  of  occupation-specific  descriptors  in  a comprehensive  occupational  information 
system  such  as  0*NET.  The  inclusion  of  occupation-specific  information,  however,  raises  a host 
of  issues.  As  illustrated  in  the  history  of  the  Dictionary  of  Occupational  Titles  (DOT;  U.S. 
Department  of  Labor,  1991)  and  the  concerns  voiced  by  the  Advisory  Panel  for  the  DOT 
(APDOT,  1993),  occupation-specific  information  often  is  collected  in  a patchwork  fashion. 
Further,  by  its  very  nature,  this  kind  of  descriptive  information  is  difficult  to  embed  within  a 
broader  organizing  structure.  Finally,  collection  of  occupation-specific  descriptive  information  is 
resource  intensive. 

These  and  other  considerations  suggest  that  it  may  be  difficult  and  perhaps  not  especially 
useful  to  include  occupation-specific  information  in  the  0*NET.  At  least  over  the  short  run  there 
seems  to  be  some  truth  to  this  proposition.  Certainly,  there  is  little  point  in  collecting  volumes  of 
occupation-specific  information  until  the  various  cross-occupation  descriptors  have  been  used  to 
identify  a coherent  and  reasonably  parsimonious  set  of  occupations  and  job  families.  Once  that 
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has  been  done,  however,  and  once  a viable  framework  for  collecting  occupation-specific 
descriptors  has  been  constructed,  then  this  more  specific  type  of  descriptive  data  should  be 
included  in  a truly  comprehensive  occupational  information  system. 

Occupation-Specific  Tasks 
Background 

As  Fleishman  and  Quaintance  (1984)  point  out,  a variety  of  techniques  might  be  used  to 
generate  descriptive  information  about  the  activities  being  performed  on  a particular  job.  One 
might,  for  example,  describe  occupations  in  terms  of  performance  errors.  Alternatively,  the  work 
people  do  in  their  jobs  can  be  described  through  qualitative  ethnographic  procedures.  Still 
another  approach  to  the  description  of  occupational  activities  flows  from  recent  work  on  the  role 
of  cognition  in  performance.  In  this  instance,  activities  are  described  in  terms  of  the  differences 
observed  in  the  knowledge  structures  characterizing  experts  and  novices  in  a particular  field  of 
endeavor  (Camara,  1992). 

Each  of  these  techniques  used  to  describe  the  requirement  of  a given  job  have  their 
unique  strengths  and  weaknesses.  Further,  it  should  be  apparent  that  these  techniques  all  provide 
somewhat  different  information  about  the  nature  of  performance  on  a certain  job.  By  the  same 
token,  however,  these  techniques  all  share  a common  starting  point  in  that  they  begin  with  an 
attempt  to  define  the  nature  of  the  activities  people  are  performing  on  their  jobs. 

As  a result,  the  definition  and  description  of  work  activities  provides  a basis  for  virtually 
all  job  analysis  efforts.  The  procedure  most  commonly  used  to  define  and  describe  these 
activities  is  task  analysis.  At  a general  level,  task  analysis  represents  a way  of  framing  or 
organizing  job  activities.  As  McCormick  (1979a)  points  out,  fundamentally,  a task  is  an  action 
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applied  to  some  object  under  certain  conditions.  The  starting  point  for  attempts  to  describe  a 
specific  occupation  is,  therefore,  definition  of  the  tasks  to  be  performed  on  the  occupation. 

A variety  of  procedures  have  been  used  to  identify  the  tasks  performed  in  an  occupation 
and  to  obtain  descriptive  information  about  the  nature  of  requisite  job  tasks.  Tasks  are  sometimes 
identified  by  having  occupational  analysts  watch  people  perform  the  work.  At  other  times, 
individual  interviews  or  panel  meetings  with  incumbents  or  supervisors  are  used  to  define  tasks. 
A third  approach  is  to  use  existing  task  inventories  (McCormick,  Jeanneret,  & Mecham,  1972), 
which  have  usually  themselves  been  developed  through  one  of  the  two  just-mentioned  methods. 
Generally,  these  task  lists  are  used  to  identify  the  more  important  or  frequently  performed  tasks 
(Friedman,  1990,  Harvey  & Lozada-Larsen,  1988).  Often,  however,  these  task  lists,  inventories, 
and  questionnaires  request  other  types  of  information  about  the  nature  of  task  performance,  such 
as  learning  difficulty,  criticality,  or  degree  of  discretion  (Mumford,  Weeks,  Harding,  & 
Fleishman,  1987). 

Although  a variety  of  procedures  may  be  used  to  identify  requisite  tasks  in  an  occupation, 
the  most  common  procedure  is  to  convene  a meeting  of  subject  matter  experts  (SMEs).  In  SME 
meetings,  a group  of  five  to  seven  incumbents,  or  their  supervisors,  are  called  together.  These 
SMEs  are  then  asked  to  describe  the  activities  performed  on  the  job  at  hand.  This  unstructured 
recall  approach  will  elicit  usable  task  statements.  However,  to  attain  comprehensive  coverage  of 
the  tasks  performed  in  an  occupation,  it  often  is  necessary  to  conduct  a number  of  meetings.  As  a 
result,  this  technique  often  becomes  unduly  time  consuming  and  expensive  (C.  Walker,  personal 
communication,  April  1995).  Alternatives  such  as  critical  incident  analysis  or  observation  of  job 
performance  also,  unfortunately,  suffer  from  much  the  same  problem. 
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This  task  identification  problem  is  important  with  respect  to  the  development  of  the 
occupational  classification  and  information  system.  Even  with  a well  developed  classification 
system,  there  still  are  likely  to  be  a relatively  large  number  of  job  families  and  occupations  within 
these  families  where  detailed  information  will  be  needed  concerning  the  underlying  work 
activities.  In  the  following  section,  we  review  four  potential  techniques  for  collecting  occupation- 
specific  information  about  the  tasks  that  are  performed,  all  of  which  might  provide  a more 
economical  and  efficient  basis  for  the  collection  of  this  descriptive  information. 

Approaches 

General  Task  Inventories 

One  approach  to  efficient  collection  of  task  data  is  the  general,  cross-occupation  task 
inventory.  Two  advantages  of  this  approach  are  that  it  is  relatively  fast  and  it  is  inexpensive. 
Perhaps  the  most  frequently  used  general  job  analysis  inventory  of  this  sort  is  the  Position 
• Analysis  Questionnaire  (PAQ;  McCormick,  Jeanneret,  & Mecham,  1972).  Cunningham  (1988) 
and  Harvey  (1991)  have  also  contributed  general  inventories.  More  recently,  the  American 
College  Testing  Program  has  developed  a general  task  survey  under  the  auspices  of  the  U.S. 
Department  of  Labor  (American  College  Testing,  1994). 

There  is  evidence  that  these  kinds  of  general  cross-occupation  task  surveys  can  provide 
meaningful  descriptions  of  activities  performed  in  a number  of  occupations  (McCormick, 
Jeanneret,  & Mecham,  1972).  On  the  other  hand,  by  virtue  of  their  focus  (i.e.,  tasks  appearing  in 
multiple  occupations),  they  necessarily  give  less  attention  to  the  specific  activities  occurring  in  a 
given  occupation  (Levine,  1983).  As  a result,  it  is  questionable  whether  this  approach  will 
provide  a truly  comprehensive  description  of  tasks  that  make  a given  occupation  unique.  This 
potential  shortcoming  of  generic  surveys  in  turn  limits  their  value  in  defining  occupation-specific 
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skills  and  knowledges.  They  may  have  greater  value,  however,  in  identifying  intermediate  work 
activities  of  the  sort  described  by  Cunningham,  Drewes,  and  Powell  (1995). 

Available  Task  Inventories 

A second  technique  that  could  be  used  to  obtain  information  about  tasks  relies  on  the  use 
of  existing  job  analysis  inventories.  Prior  efforts  have  provided  task  analysis  data  for  a number  of 
occupations.  Conceivably,  an  effort  might  be  initiated  in  which  a library  is  built  up  over  time  that 
describes  the  tasks  identified  in  earlier  job  analysis  efforts.  When  attempting  to  describe  a 
particular  occupation,  relevant  task  inventories  would  be  drawn  from  this  library.  These  task  lists 
would  be  reviewed,  and  then  used  as  the  basis  for  developing  task  inventories  tailored  to  the  job 
at  hand  that  would  presumably  extend  across  locations  or  establishments. 

This  approach  clearly  would  reduce  the  amount  of  effort  needed  to  generate  an  initial  set 
of  task  statements.  However,  a substantial  amount  of  work  would  be  required  to  develop  the  kind 
of  library  needed  to  make  this  approach  feasible.  Available  task  analysis  inventories  would  have 
to  be  rewritten  so  that  task  statements  were  written  at  a common  level  of  detail  that  would  indeed 
permit  generalization  across  establishments.  Further,  arrangements  would  have  to  be  made  to 
obtain  what  in  many  areas  would  be  proprietary  data.  Finally,  available  task  inventories  are  likely 
to  be  available  only  for  a relatively  select  sample  of  occupations—typically  occupations  in  which 
performance  is  of  sufficient  importance  to  the  organization  to  warrant  an  expensive  and  time 
consuming  job  analysis  effort.  As  a result,  other  techniques  would  still  be  required  for  identifying 
the  tasks  occurring  on  other  occupations  not  covered  by  the  available  task  inventories. 

Even  if  these  difficulties  could  be  overcome,  one  problem  remains:  The  lists  of  tasks 
would  probably  be  deficient.  There  are  at  least  three  reasons  for  this  potential  deficiency:  (1) 

r, 

some  of  the  inventories  would  be  based  on  relatively  old  job  analyses  and  important  changes  in 
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the  occupations  may  have  occurred  since  the  inventories  were  developed;  (2)  some  of  the 
inventories  would  be  based  on  job  analyses  that  were  performed  for  specific  purposes,  resulting 
in  inventories  focusing  only  on  some  parts  of  the  examined  occupations;  and  (3)  the  inventories 
would  be  based  on  job  analyses  that  vary  greatly  in  quality  and  comprehensiveness. 

Activity  Analysis 

A third  approach  for  generating  occupation-specific  tasks  in  rapid,  cost  effective  fashion 
has  been  suggested  by  Prien  (1994).  This  approach  is  based  on  the  notion  that  all  occupations 
involve  a limited  set  of  common  activities,  such  as  installing,  repairing,  writing,  tracking,  or 
supervising.  These  common  activities,  or  action  verbs,  might  be  used  as  a basis  for  generating 
tasks  by  applying  the  following  procedures.  First,  a general  taxonomy  of  action  verbs  would  be 
identified,  such  as  those  proposed  by  Prien  (1994)  or  Edwards  (1989).  Second*  a group  of  SMEs 
would  be  asked  to  review  this  activity  list  and  check  the  activities  they  perform  in  their 
occupation.  Third,  a second  group  of  SMEs  would  be  presented  with  each  of  these  activities  and 
for  each  activity  they  would  be  asked  to  list  all  objects  of  the  activity,  providing  modifications  as 
necessary.  Thus,  in  the  case  of  electricians,  incumbents  might  list  “install  switches,”  “install 
wiring,”  and  “install  control  boxes.” 

The  procedures  described  above  would,  at  least  in  theory,  result  in  a set  of  task  statements 
consistent  with  McCormick's  (1979b)  definition  of  a task  as  an  activity  occurring  in  relation  to 
some  object.  Further,  it  is  possible  that  this  procedure,  if  coupled  with  an  expert  system,  might  be 
used  to  generate  tasks  "on-line"  or  through  telephone  interviews.  Thus,  an  activity-based 
approach  to  task  generation  would  seem  to  warrant  further  consideration. 

We  use  the  word  "consideration”  quite  intentionally.  Although  this  activity-based 
approach  to  task  generation  has  some  attractive  features,  it  has  not  been  widely  applied. 
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Moreover,  it  may  prove  difficult  to  obtain  a comprehensive  list  of  activity  statements  and  create 
procedures  for  identifying  synonyms  within  a list  of  activity  statements.  Additionally,  any 
activity-based  approach  may  implicitly  downplay  the  specific  objects  and  the  unique  behavioral 
demands  associated  with  a given  set  of  tasks. 

Generalized  Work  Activities 

A fourth  approach  that  might  be  used  to  generate  task  statements  has  been  suggested  by 
Mumford  and  his  colleagues  (Clifton,  Connelly,  Reiter-Palmon,  & Mumford,  1991;  Connelly, 
Reiter-Palmon,  Clifton,  & Mumford,  1991;  Gilbert,  Connelly,  Clifton,  Reiter-Palmon,  & 
Mumford,  1992;  Mumford,  Threlfall,  Costanza,  Baughman,  & Smart,  1992;  Reiter-Palmon,  et 
al.,  1990).  In  this  approach,  a panel  of  SMEs  is  presented  with  a list  of  generalized  work 
activities  (GW As),  like  those  described  in  Chapter  6.  Initially,  panel  members  are  asked  to  reach 
a consensus  regarding  the  GW As  for  their  occupations.  Finally,  within  each  of  the  retained 
GW  As,  panel  members  are  asked  to  list  the  specific  tasks  occurring  on  their  occupation. 

This  approach  differs  from  more  traditional  approaches  in  the  process  used  to  elicit  tasks. 
Rather  than  being  asked  a global  question,  "What  do  you  do  on  your  job?,"  panel  members  are 
asked  a series  of  more  specific  questions  referring  to  the  tasks  performed  under  a given 
dimension  or  GW  A.  By  using  recall  in  relation  to  GW  As,  tasks  can  be  generated  far  more 
rapidly.  Typically,  the  research  cited  above  showed  that  a one-day  panel  mating  with  five  or  six 
SMEs  is  sufficient  to  obtain  90%  to  95%  coverage  of  the  occupation's  relevant  tasks. 

Perhaps  the  most  important  characteristic  of  this  approach  is  that  it  allows  tasks  to  be 
generated  quickly  with  good  coverage  of  the  relevant  domain.  Two  other  characteristics  of  the 
approach  also  make  it  attractive.  First,  it  can  be  easily  extended  to  capture  contextual  influences 
on  performance  as  well  as  duties  and  tools.  Second,  it  allows  tasks  to  be  generated  and  organized 
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within  broader  GW  As,  thereby  providing  a system  for  identifying  tasks  that  explicitly  integrates 
these  tasks  into  a broader  taxonomic  structure.  In  terms  of  resources  required,  this  approach 
should  be  more  expensive  and  time-consuming  than  the  general  task  inventory,  but  less 
expensive  than  the  library  approach. 

Procedures  and  Findings 

Our  discussion  of  job  analysis  procedures  suggests  a two-step  approach  to  the 
development  of  a task  database  for  inclusion  in  0*NET.  Initially,  for  the  purposes  of  the 
prototype,  a variant  of  the  Available  Task  Inventories  approach  described  above  was  used. 
Subsequently,  however,  the  GWA  approach  might  be  employed  or,  alternatively,  an  activity 
analysis  approach.  Both  approaches  are  described  below. 

A Modified  Available  Task  Inventories  Approach 

A variation  on  this  library-based  task  generation  procedure  was  employed  for  the 
prototype  0*NET.  The  current  Dictionary  of  Occupational  Titles  (DOT)  provides  a short  list  of 
major  tasks  to  be  performed  in  each  occupation  included  in  the  DOT.  Thus,  it  is  possible  to 
identify  a set  of  tasks  by  abstracting  core  tasks  included  in  those  occupations  subsumed  under  a 
given  OES  occupation. 

This  work  was  performed  by  the  North  Carolina  Occupational  Analysis  Field  Center 
(OAFC).  The  OAFC  began  by  identifying  those  DOT  occupations  associated  with  each  of  the  80 
OES  occupations  targeted  in  the  initial  0*NET  data  collection  effort.  This  initial  clustering 
served  to  convert  DOT  occupations  into  the  broader  OES  occupational  structure.  Next,  a text 
analysis  program  was  used  to  abstract  tasks  from  the  relevant  DOT  occupational  descriptions. 

Once  the  tasks  occurring  within  a cluster  of  DOT  occupations  had  been  identified,  they 
were  presented  to  analysts  at  the  OAFC  field  centers.  These  experienced  occupational  analysts 
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then  edited  the  identified  tasks  for  clarity,  generality,  brevity,  and  accuracy.  Additionally,  a 
second  panel  of  occupational  analysts  reviewed  the  resulting  list  of  task  statements  for  generality 
and  comprehensiveness,  adding  to  or  correcting  the  initial  task  list  to  ensure  that  the  task  list  was 
reasonably  comprehensive  and  that  the  tasks  were  written  to  a common  level  of  specificity. 

This  procedure  resulted  in  the  identification  of  7 to  30  relatively  broad  task  statements  for 
each  of  the  80  targeted  OES  occupations.  The  task  lists  were  then  used  to  create  occupation- 
specific  task  rating  questionnaires.  One  of  these  questionnaires  is  presented  in  Figure  16-1.  As 
shown  in  Figure  16-1,  job  incumbents  were  asked  to  read  each  task  statement  and  identify  those 
tasks  relevant  to  the  performance  of  their  job.  If  incumbents  indicated  that  a task  is  relevant,  they 
were  asked  to  (a)  rate  the  frequency  with  which  they  perform  this  task  and  (b)  rate  the  importance 
of  this  task  with  respect  to  performance  of  the  job.  Importance  and  frequency  ratings  are  used  in 
part  because  they  have  been  shown  to  provide  unique  descriptive  information  and  in  part  because 
the  simultaneous  collection  of  multiple  ratings  contributes  to  reliability  (McCormick,  1979b).  It 
is  of  note  that  frequency  ratings  were  collected  on  an  absolute,  rather  than  a relative,  scale. 

The  tasks  shown  in  Figure  16-1  represent  a modified  version  of  the  Available  Task 
Inventories  notion  described  earlier  in  this  chapter.  In  a more  elaborate  version,  the  initial  task 
lists  would  come  from  multiple  sources  (i.e.,  not  just  the  DOT  task  lists).  They  would  stem  from 
a review  of  prior  job  analysis  programs,  including  the  job  analysis  work  conducted  by  (a)  the 
Department  of  Defense,  (b)  the  Office  of  Personnel  Management,  and  (3)  those  research 
institutes,  consulting  firms,  and  employers  willing  to  share  proprietary  data.  The  process  would 
include  a careful  review  of  the  quality  of  each  job  analysis  and  its  resulting  task  list.  The  tasks 
list(s)  that  are  judged  of  sufficient  quality  would  be  (1)  edited  for  clarity  and  a common  level  of 
specificity,  (2)  reviewed,  (3)  and  edited  again  as  necessary.  It  is  important  to  note  that  this 
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procedure  would  likely  result  in  occupation-specific  task  rating  questionnaires  with  a greater 
number  of  tasks  per  occupation  (e.g.,  100  - 200),  and  at  a more  specific  level  than  tasks  derived 
from  the  DOT. 

Bearing  these  caveats  in  mind,  the  results  obtained  in  the  0*NET  initial  data  collection 
do  provide  support  for  the  potential  value  of  the  archival  approach.  Job  incumbents  were  asked  to 
complete,  in  addition  to  between  one  and  five  of  the  questionnaires  described  in  the  other 
chapters  of  this  report,  an  occupation-specific  task  questionnaire  (Figure  16-1  provides  an 
example).  Table  16-1  presents  the  interrater  agreement  coefficients  obtained  for  task  frequency 
and  importance  ratings.  In  addition  to  the  observed  interrater  agreement  coefficients,  estimates 
for  1 and  for  30  raters  are  also  presented.  The  interrater  agreement  coefficients,  reflecting  the 
degree  of  consistency  across  judges  in  their  task  ratings  within  an  occupation,  are  uniformly  high, 
ranging  from  .80  to  .98  for  the  frequency  rating  and  .77  to  .97  for  the  importance  rating,  for  30 
incumbents.  The  occupations  with  the  lowest  reliabilities— Stock  Clerks  and  Packaging  and 
Filling  Machine  Operators— may  be  those  with  the  least  consistency  in  task  demands  across,  or 
even  within,  organizations. 

Table  16-2  presents  interrater  agreement  coefficients  where  task  ratings  were  scored  in 
three  ways:  (1)  using  the  full  frequency  and  importance  scales,  as  in  Table  16-1;  (2)  using  only 
relevant  responses  on  the  frequency  and  importance  scales  (not  relevant  responses  set  to 
missing);  and  (3)  dichotomously  (relevant/not  relevant).  While,  predictably,  the  relevant-only  and 
dichotomous  rescoring  yielded  lower  reliabilities  than  the  full  scale  scoring,  the  reliabilities  for 
these  two  rescorings  are  very  similar  to  full-scale  scoring  and  surprisingly  high.  With  the 
exception  of  incumbents  in  the  Packaging  and  Filling  Machine  Operators  occupation,  a large 
number  of  whom  indicated  that  the  tasks  on  the  occupation-specific  questionnaire  were  not 
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relevant  for  their  jobs,  the  relevant-only  reliabilities  for  the  frequency  scale  ranged  from  .74  to 
.99  and  the  relevant/not  relevant  reliabilities  ranged  from  .66  to  .99.  This  finding  provides  strong 
support  for  the  inclusion  of  the  not  relevant  response  option. 

Additional  evidence  bearing  on  the  meaningfulness  of  these  descriptions  might  be 
obtained  by  considering  the  relationships  observed  between  the  frequency  and  importance 
ratings.  In  accordance  with  the  results  obtained  in  earlier  studies  (Harvey,  1990),  the  task 
frequency  and  importance  ratings  yielded  the  expected  positive  relationship.  As  reported  in  Table 
16-3,  across  occupations,  the  correlations  between  task  frequency  and  importance  ratings  ranged 
from  .55  to  .92,  with  a median  correlation  of  .78.  These  results  suggest  that  incumbents  are 
differentiating  between  the  frequency  and  importance  scales.  The  relatively  short  length  of  the 
occupation-specific  task  questionnaires,  the  greater  job  relevance  of  the  questions,  and  the  fact 
that  incumbents  may  have  had  prior  experience  with  similar  job  analysis  instruments,  all  may 
have  contributed  to  increased  engagement  with  the  rating  tasks  on  the  task  questionnaires  and, 
consequently,  well-differentiated  frequency  and  importance  ratings. 

Although  the  judges  clearly  could  agree  with  respect  to  how  well  these  ratings  described 
their  jobs  and  could  differentiate  between  the  frequency  and  importance  aspects  of  the  task 
ratings,  these  results  do  not  provide  a great  deal  of  evidence  bearing  on  the  meaningfulness  of 
these  ratings  in  describing  people's  jobs.  One  way  to  assess  the  meaningfulness  of  this  descriptive 
data  is  by  examining  these  task  ratings  with  respect  to  the  nature  of  certain  occupations. 

Table  16-4  presents  the  means  and  standard  deviations  of  incumbents’  frequency  and 
importance  ratings  for  two  of  the  29  occupations  examined  here—General  Managers  and  Top 
Executives,  and  Police  Patrol  Officers.  The  results  for  these  two  occupations  are  fairly  typical 
and  illustrate  the  types  of  rating  patterns  evidenced  in  these  data.  For  General  Managers  and  Top 
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Executives,  for  example,  crucial  job  duties,  such  as  "Directs  and  coordinates  activities  of  workers 
to  ensure  continuing  operations,  maximize  returns  on  investments,  and  increase  productivity" 
(Importance:  M = 4.05,  SD  = 1.02)  and  "Consults  with  staff  to  assist  in  making  decisions  and 
coordinating  activities"  (Importance:  M = 4.00,  SD  = .82)  tended  to  receive  relatively  high  mean 
importance  ratings.  Mean  frequency  ratings  also  provide  some  evidence  for  the  meaningfulness 
of  this  descriptive  information.  For  example,  it  is  hardly  surprising  that  Police  Patrol  Officers 
tended  to  spend  more  time  on  the  task  "Patrols  specific  area  on  foot,  horseback,  or  motorized 
conveyance"  (Frequency:  M = 5.75,  SD  = 1.75)  than  on  "Expedites  processing  of  prisoners, 
prepares  and  maintains  records  of  prisoner  bookings,  and  maintains  record  of  prisoner  status 
during  booking  and  pre-trial  process"  (Frequency:  M = 0.86,  SD  = 1.70). 

Somewhat  more  compelling  evidence  for  the  meaningfulness  of  these  tasks  might  be 
obtained  by  comparing  those  descriptions  with  other,  independent  measures  of  people’s  job 
activities.  Accordingly,  tasks  for  ten  occupations  were  classified  into  the  GW  As.  The  initial 
assignments  were  made  by  a single  judge,  but  were  confirmed  by  other  occupational  analysts. 

The  assignments  are  reported  in  Figure  16-2.  This  figure  also  presents  the  correlations  of  task  and 
GWA  ratings  and  the  percentage  agreement  in  relevant/not  relevant  classification.  The  generally 
low  correlations  between  the  frequency  ratings  for  the  tasks  and  corresponding  GW  As,  and 
between  the  importance  ratings,  are  most  likely  due  to  a difference  in  level  of  specificity  between 
the  descriptors.  For  example,  a task  classified  into  a GWA  might  nevertheless  constitute  a very 
small  part— or  a very  specialized  application—of  that  GWA.  The  percentage  agreement  between 
the  ratings  of  the  tasks  and  GW  As  in  terms  of  whether  that  descriptor  is  relevant  for  the  job  is 
more  indicative  of  the  consistency  of  the  ratings.  In  fact,  these  percentages  tended  to  be  quite 
substantial,  with  the  majority  of  task-GWA  pairs  showing  at  least  a 70%  agreement. 
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These  findings  provide  some  evidence  for  the  potential  meaningfulness  of  task  statements 
generated  using  this  archival  approach.  On  the  other  hand,  however,  it  should  be  recognized  that 
archival  data  may  at  times  grow  dated.  Further,  this  approach  may  not  yield  viable  data  when  our 
concern  is  new  or  rapidly  changing  jobs.  Thus,  there  is  a need  for  alternative  strategies  that  might 
be  used  in  generating  occupational  tasks. 

The  GWA  Approach 

The  procedures  sketched  out  above  are  most  appropriate  when  existing  task  inventories 
are  available.  However,  a different  approach  will  need  to  be  applied  when  available  up-to-date 
task  lists  are  not  available.  As  noted  above,  the  most  attractive  approach  for  generating  these  new 
task  lists  is  to  use  a cued  recall  approach  where  the  GW  As  provide  a basis  for  task  generation. 

Broadly  speaking,  this  approach  is  based  on  the  earlier  work  of  Mumford  and  his 
colleagues  (Mumford  & Supinski,  1995;  Mumford,  Threlfall,  Costanza,  Baughman,  and  Smart, 
1992;  Reiter-Palmon,  et  al.,  1990),  although  the  general  techniques  (e.g.,  using  SME  panels)  are 
broadly  applied  in  job  analysis.  This  task  generation  procedure  requires  a group  of  five  to  six 
SMEs.  Typically,  these  SMEs  are  incumbents  or  supervisors  who  have  at  least  six  months 
experience  in  a job.  It  generally  is  useful  when  forming  SME  panels  to  select  panel  members 
who  have  different  backgrounds  and  somewhat  different  career  histories  (Campion,  1992;  Landy 
& Vasey,  1991).  Additionally,  panel  members  should  be  good  performers  who  hold  roughly 
similar  positions  in  the  organization  (Landy  & Vasey,  1991).  It  may  also  be  desirable  to  select 
panel  members  to  represent  different  organizations,  if  this  is  possible,  when  there  is  a need  to 
obtain  task  data  that  extend  across  organizations.  Alternatively,  multiple  panel  meetings  could  be 
conducted.  The  panel  meetings  should  be  scheduled  to  occur  over  a day  and  a half.  However,  all 
requisite  exercises  typically  can  be  completed  in  a single  day. 
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The  procedures  used  to  generate  task  statements  in  these  meetings  are  quite 
straightforward.  First,  each  subject  matter  expert  is  asked  to  review  the  definition  of  each  GWA 
and  determine  whether  this  type  of  activity  is  performed  on  their  job.  Second,  panel  members  are 
asked  to  reach  a consensus  decision  concerning  the  concerning  the  approximately  10  to  15 
GW  As  that  represent  the  most  important  components  of  job  performance.  Panel  members  are 
asked  to  (1)  identify  the  major  subcategories  of  activities  falling  under  this  general  dimension 
and  (2)  then  specify  the  specific  tasks  falling  under  each  subcategory.  Figure  16-3  provides  the 
instructions  given  for  the  task  generation  portion  of  the  exercise.  Essentially,  those  instructions 
ask  each  panel  member  to  list  the  major  kinds  of  tasks  they  perform  under  each  GWA  and  then 
list  the  more  discrete  tasks  subsumed  under  a given  area. 

After  panel  members  have  generated  their  lists  of  activities  and  specific  tasks  under  each 
GWA,  one  panel  member  is  asked  to  read  aloud  one  task  type  and  the  associated  tasks  he  or  she 
generated.  Other  panel  members  then  review  these  task  statements  for  relevance, 
comprehensiveness,  and  clarity,  recommending  any  necessary  changes.  This  procedure  is 

o 

repeated  round-robin,  until  panel  members  feel  that  all  work  activities  falling  under  a given 
GWA  have  been  exhausted,  and  then  the  group  moves  to  the  next  GWA.  Throughout  this 
process,  occupational  analysts  take  notes,  writing  down  the  task  statements  proposed  by  panel 
members.  After  the  end  of  the  meeting,  or  meetings,  those  notes  are  used  to  develop  the  final  task 
list. 

Typically,  these  procedures  provide  a rather  detailed  description  of  people's  jobs,  yielding 
150  to  400  tasks,  depending  on  the  complexity  of  the  job.  Figure  16-4  provides  an  illustration  of 
the  kind  of  tasks  generated  using  this  technique.  The  interrater  agreement  coefficients  obtained 
when  incumbents  were  asked  to  assess  the  frequency  with  which  these  tasks  are  performed  on 
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their  job  and  the  importance  of  these  tasks  were  similar  to  those  obtained  for  the  archival  tasks 
described  earlier.  In  some  applications  of  this  method,  a sample  of  five  occupations,  including 
sales  representatives,  managers,  finance  analysts,  telecommunications  repair  technicians  and 
telecommunications  systems  operators,  the  median  interrater  agreement  coefficient  for  frequency 
rating  was  .88  for  samples  of  twenty  judges  (Mumford  & Supinski,  1995;  Mumford,  Threlfall, 
Costanza,  Baughman,  & Smart,  1992;  Reiter-Palmon,  et  al.,  1990).  For  importance  ratings,  the 
median  interrater  agreement  coefficient  was  .91. 

These  interrater  agreement  coefficients  are  noteworthy  for  two  reasons.  First,  the  tasks 
resulting  from  the  "top-down"  GWA  approach  apparently  provide  reasonably  reliable 
descriptions  of  people's  job  activities.  Second,  it  appears  that  this  procedure  is  a reasonably 
flexible  one  in  the  sense  that  it  yields  viable  task  statements  across  a range  of  occupations 
including  sales  representatives  and  technicians  as  well  as  managers  and  financial  analysis. 

These  observations  about  the  reliability  of  task  ratings  bring  to  the  fore  a new  question:  Is 
there  reason  to  suspect  that  this  procedure  yields  task  ratings  that  provide  a meaningful 

o 

description  of  people's  jobs?  One  piece  of  evidence  bearing  on  the  meaningfulness  of  these  task 
statements  has  been  provided  by  a series  of  “comprehensiveness”  tests  that  were  part  of  the 
studies  cited  above.  In  those  tests,  three  to  five  subject  matter  experts  were  asked  to  review  the 
lists  of  tasks  obtained  using  these  procedures  and  identify  any  additional  tasks  needed  to  ensure  a 
comprehensive  description  of  people's  jobs.  Although  this  comprehensiveness  test  often  results 
in  the  addition  of  a few  tasks,  the  number  of  tasks  added  was  relatively  small  and  in  no  case 
resulted  in  more  than  a 3%  increase  in  the  number  of  tasks  identified.  Thus,  it  appears  that  this 
procedure  in  fact  yields  the  sort  of  tasks  which  provide  a comprehensive  description  of  people's 
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Another  way  one  might  assess  the  meaningfulness  of  the  descriptive  information 
provided  by  these  procedures  is  by  examining  the  ability  of  these  task  ratings  to  capture  cross- 
position  and  cross-site  differences  in  the  nature  of  the  work  being  done.  For  example,  in  the 
Reiter-Palmon,  et  al.  (1990)  study  it  was  found  that  the  resulting  task  statements  accurately 
identified  changes  in  work  load  associated  with  new  shipping  and  pricing  procedures.  The 
Mumford  and  Supinski  (1995)  study  contrasted  the  tasks  required  in  emerging  and  diminishing 
telecommunications  positions.  As  might  be  expected  based  on  the  changes  in  technology 
occurring  in  the  telecommunications  field,  tasks  associated  with  "black  box"  technology  were 
found  to  be  more  important  and  more  frequently  performed  in  emerging  as  opposed  to 
diminishing  positions. 

Taken  as  a whole,  these  studies  have  provided  some  compelling  evidence  for  the 
reliability  and  validity  of  task  statements  formulated  using  this  top-down  GWA  task  generation 
procedure.  This  task  generation  procedure,  however,  has  been  found  to  display  two  other 
noteworthy  characteristics.  First,  it  appears  to  provide  an  unusually  efficient  basis  for  task 
generation.  Typically,  two  one-day  meetings  are  sufficient  to  obtain  a set  of  comprehensive, 
detailed  task  statements.  This  greater  efficiency  of  the  cued  recall  strategy  being  used  in  task 
generation,  of  course,  greatly  reduces  the  cost,  as  well  as  the  time,  needed  to  obtain  occupation- 
specific  tasks.  Second,  because  tasks  are  generated  and  organized  in  terms  of  the  GWAs,  it 
becomes  possible  to  integrate  occupation-specific  tasks  into  a larger  taxonomy  (i.e.,  the  GWA 
portion  of  the  0*NET  taxonomy).  The  GWAs  also  provide  a basis  for  organizing  fask-  data  in  a 
more  user-friendly  format.  Finally,  as  indicated  in  the  following  sections,  this  framework  for  task 
generation  is  an  unusually  flexible  one,  allowing  it  to  be  extended  to  incorporate  elements  of  an 
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activity  analysis  approach  permitting  the  "on  line"  identification  of  job  tasks  through  an  artificial 
intelligence  system. 

The  fundamental  notion  underlying  activity  analysis  is  that  tasks  can  be  identified  and 
structured  with  respect  to  their  two  key  components.  In  other  words,  tasks  can  be  defined  with 
respect  to  the  action  being  taken-a  verb— and  the  thing  being  manipulated  in  this  action-an 
object  (Gael,  1983).  At  a general  level,  it  is  difficult  to  dispute  this  argument;  certainly,  verbs  and 
objects,  along  with  object  and  verb  modifiers,  represent  the  key  linguistic  components  of  any  task 
statement.  On  the  other  hand,  lists  of  verbs  and  objects  have  proven  of  limited  value  simply 
because  there  are  so  many  verbs  and  objects  used  to  describe  work  activities  in  the  English 
language. 

On  the  other  hand,  however,  the  generalized  work  activities  strategy  for  task  generation 
suggests  a way  that  one  might  employ  these  general  characteristics  of  job  tasks  in  using  activity 
analysis  to  develop  task  inventories.  Any  given  generalized  work  activity— for  example,  operating 
heavy  equipment— will  have  associated  with  it  a limited  set  of  verbs,  (e.g.,  drive,  control)  and 
objects  (e.g.,  trucks,  shovels).  Thus,  by  organizing  verbs  and  objects  in  terms  of  the  associated 
generalized  work  activities,  it  might  be  possible  to  use  this  information  in  developing  an 
automated  job  analysis  system  intended  to  facilitate  task  analysis  efforts  (see,  for  example. 
Appendix  16- A). 

Such  a system  might  operate  as  illustrated  in  Figure  16-5.  Task  generation  would  be  done 
by  presenting  a generalized  work  activity  along  with  a list  of  (1)  the  verbs  most  frequently 
associated  with  this  generalized  work  activity  and  (2)  the  objects  most  frequently  associated  with 
this  generalized  work  activity.  Incumbents  would  be  asked  to  review  this  initial  list  and  add  any 
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new  verbs  or  objects  associated  with  a generalized  work  activity  held  to  be  relevant  to  describing 
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their  jobs.  Next,  they  would  begin  constructing  task  statements  by  matching  particular  verbs  with 
particular  objects  to  create  a prototype  job  task  (e.g.,  drive  trucks).  After  a prototype  task  had 
been  constructed  thropgh  these  verb-object  matches,  the  incumbent  would  then  be  asked  to 
modify  the  object  statement  (drive  trucks  to  parcel  pickup  sites)  or  the  verb  statement  (drive 
delivery  trucks  to  parcel  pickup  sites).  The  resulting  task  statements  obtained  from  10  to  15 
incumbents  would  be  written  out  or  stored  in  a computer.  Subsequently,  these  tasks  would  be 
sorted  by  generalized  work  activity,  verbs,  and  objects,  to  help  analysts  prepare  a consolidated 
list  (Task  Editing  in  Figure  16-5). 

To  test  this  approach  to  task  list  generation,  we  developed  computer  programs  to  generate 
and  edit  task  statements.  These  programs  are  part  of  a larger  set  of  occupational  analysis 
computer  tools  currently  being  developed  and  tested.  In  recent  usability  tests,  employees  used  the 
task  generation  software  to  create  occupation-specific  task  lists.  Figure  16-6  illustrates  typical 
task  statements  sampled  from  these  lists.  An  initial  pilot  test  of  the  task  editing  program  showed 
that  it  is  an  effective  tool  for  integrating  lists  of  tasks  from  multiple  incumbents.  Expert  review  of 
such  an  edited  task  list  indicated  that  the  task  generation  and  editing  procedures  yielded  a 
comprehensive  list  of  tasks  at  the  appropriate  level  of  detail.  Based  on  this  work  and  past  studies 
employing  the  GWA  approach  (e.g.,  Mumford  & Supinski,  1995),  we  predict  that  more  extensive 
analyses  of  task  lists  generated  in  this  manner  will  support  its  utility.  Preliminary  evidence  shows 
that  the  combination  of  the  GWA  approach  to  task  generation  and  the  use  of  computer-based  job 
analysis  tools  not  only  provides  valid  task  lists,  but  also  provides  significant  gains  in  efficiency 


and  cost-effectiveness. 
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Summary 


At  this  point,  it  should  be  apparent  that  the  attention  given  to  occupation-specific 
descriptors  involved  a careful  examination  of  alternative  procedures  for  identifying  and 
describing  requisite  job  tasks.  In  fact,  substantial  effort  seemed  called  for,  in  part  because  task 
data  provides  the  essential  piece  of  information  needed  to  identify  and  understand  virtually  all 
other  occupation-specific  descriptors.  Further,  occupation-specific  data,  particularly  task 
descriptions,  represent  a cornerstone  of  many  personnel  interventions,  including  training,  job 
redesign,  and  human  factors  analysis. 

Unfortunately,  however,  it  has  traditionally  proven  difficult  to  apply  task  descriptions  in 
an  effective  fashion  across  large  numbers  of  occupations.  In  fact,  this  problem  might  be 
attributed  to  the  high  cost  of  obtaining  occupation-specific  information,  particularly  task 
descriptions.  In  part  however,  the  number  of  tasks  involved  in  most  jobs  and  the  use  of  multiple 
ratings  make  it  difficult  to  apply  this  information,  because  the  resulting  data  cannot  be  readily 
organized  within  a broader  structure  that  facilitates  various  applications. 

The  approaches  described  above,  particularly  task  generation  based  on  the  generalized 
work  activities  are  expressly  intended  to  address  this  issue.  In  fact,  the  results  obtained  in  these 
initial  efforts  are  promising.  Not  only  can  the  generalized  work  activities  be  used  to  organize  task 
data,  it  appears  that  the  availability  of  this  organizing  structure  allows  us  to  gather 
comprehensive,  reliable,  and  valid  task  data  in  less  time  and  at  a lower  cost  than  more  traditional 
procedures.  Further,  at  least  in  some  cases,  further  gains  in  efficiency  might  be  obtained  using 
automated  systems,  based  on  a combination  of  the  generalized  work  activity  and  activity  analysis 
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strategies.  Thus,  there  is  good  reason  to  suspect  that  the  cross-occupation  model  may  provide  the 
infrastructure  needed  for  the  timely  and  efficient  collection  of  occupation-specific  information.1 

Occupation-Specific  Skills  and  Knowledges 

As  noted  elsewhere,  both  in  this  report  and  in  the  broader  literature,  a variety  of 
procedures  has  been  used  to  define  occupation-specific  skills.  In  some  cases,  an  occupational 
skill  is  defined  as  practiced  task  performance.  In  other  cases,  occupation-specific  skills  are 
defined  in  terms  of  broad  basic  capacities. 

In  the  case  of  occupation-specific  knowledges,  the  situation  is  somewhat  less  ambiguous. 
Most  would  agree  that  knowledge  represents  an  organized  set  of  facts  and  principles  pertaining  to 
the  characteristics  of  objects  lying  in  some  domain  (Fleishman  & Mumford,  1989).  Further,  a 
variety  of  techniques,  including  expert-novice  comparisons  (Chi,  Glaser,  and  Farr,  1988)  and 
think-aloud  protocols  (Barsalou,  1991),  are  available  for  identifying  relevant  knowledge 
structures.  Although  the  evidence  indicates  that  these  techniques  can  be  effectively  used  to 
identify  requisite  knowledges,  they  also  are  relatively  time  consuming.  In  fact,  the  time  involved 
in  applying  these  techniques  is  such  that  it  would  effectively  prohibit  use  of  this  approach  in 
defining  occupation-specific  knowledges  across  a range  of  occupations. 

Given  the  importance  of  occupation-specific  skills  and  knowledges  in  assessment, 
selection,  training  design,  and  re-skilling,  techniques  are  needed  that  will  permit  cost  effective 
collection  of  this  information.  Thus,  in  this  section,  we  sketch  out  a set  of  general  procedures  that 
might  be  used  to  identify  occupation-specific  skills  and  knowledges. 


A more  fully-developed  version  of  this  kind  of  system  implemented  with  computer  assistance  is  described  in 
Appendix  16- A. 
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Skills  and  Knowledges 

Before  proceeding  to  the  procedures  that  will  be  used  to  identify  occupation-specific 
skills  and  knowledges,  it  would  be  prudent  to  consider  exactly  what  is  meant  by  the  terms  skills 
and  knowledges.  As  described  in  Chapter  3,  skills  have  been  defined  in  a variety  of  different 
ways.  One  way  to  define  skills  is  as  a general  set  of  activities  or  procedures  required  for 
performance  in  some  domain  (Campbell,  McCloy,  Oppler,  & Sager,  1992).  Depending  on  the 
breadth  of  the  domain,  skills  can  be  defined  at  a number  of  different  levels.  Further,  there  may  be 
a number  of  different  types  of  skills.  For  example,  basic  skills  might  reflect  activities  needed  to 
learn  various  tasks.  Cross-functional  skills  might  represent  general  activities  or  procedures  called 
for  in  domains  that  extend  across  occupations. 

Occupation-specific  skills  represent  still  another  kind  of  skill.  In  this  case,  the  domain  is 
some  subset  of  the  occupation’s  tasks  that  calls  for  a common  set  of  activities.  In  the  case  of 
electricians,  installation,  a common  job  activity,  might  be  applied  in  three  distinct  types  of  task 
domains:  (1)  outside  wiring  installation;  (2)  inside  wiring  installation;  and  (3)  installation  of 
lighting  fixtures.  Accordingly,  occupation-specific  skills  may  be  defined  as  a general  activity 
(c.g.,  installation,  repair)  as  applied  to  a similar  set  of  tasks  calling  for  related  procedures. 

Occupation-specific  knowledges  are  less  difficult  to  define.  Knowledge,  generally 
speaking,  is  held  to  reflect  an  organized  set  of  facts  and  principles  pertaining  to  the 
characteristics  of  objects  lying  in  some' domain.  In  this  sense,  knowledge,  particularly  expert 
knowledge,  can  be  viewed  as  a principle-based  organization  of  relevant  material  within  a given 
task  domain.  Thus,  the  principles  needed  to  work  with  or  apply  a given  skill  in  performing  a set 
of  related  tasks  may  be  said  to  reflect  an  occupation-specific  knowledge. 

Procedures  for  Defining  Occupation-Specific  Skills  and  Knowledges 
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Given  the  foregoing  definition  of  occupation-specific  skills  and  knowledges  it  becomes 
possible  to  envision  a four-step  process  for  identifying  them.  First,  the  core  tasks  for  an 
occupation  must  be  identified.  As  discussed  earlier  in  this  chapter,  the  GW  A and  general 
Available  Task  Inventories  and  GWA/Activity  Analysis  techniques  are  used  to  identify  tasks. 
Second,  tasks  reflecting  a common  set  of  basic  and  cross-functional  skill  requirements  would  be 
specified.  Third,  within  a given  task  set,  where  tasks  are  drawn  from  a common  skill,  tasks 
involving  similar  or  related  procedures  would  be  identified.  Fourth,  the  knowledges  needed  to 
perform  a given  set  of  procedurally  related  tasks  would  be  identified.  Figure  16-7  illustrates  these 
four  basic  steps. 

The  general  procedures  sketched  out  in  Figure  16-7  are  intended,  in  part,  to  address  a 
crucial  problem  identified  by  Stevens  and  Campion  (1994).  They  found  that  SMEs  typically 
group  tasks  together  based  on  content  similarity  rather  than  underlying  skill  and  knowledge 
requirements.  Accordingly,  we  would  not  ask  occupational  members  or  SMEs  to  identify 
occupation-specific  skills  directly.  Instead,  an  occupational  analyst  would  review  the  task 
statements  generated  by  these  SMEs  and  assign  tasks  to  the  most  relevant  basic  and  cross- 
functional  skills  (represented  by  the  second  box  in  Figure  16-7).  Then,  SMEs  would  define 
occupation-specific  skills  by  only  looking  at  those  tasks  held  to  represent  instances  of  one  basic 
or  cross-functional  skill  at  a time  (the  third  box  in  Figure  16-7). 

This  set  of  procedures  displays  three  desirable  characteristics.  First,  SMEs  are  presented 
with  task  sets  already  defined  in  terms  of  basic  and  cross-functional  skills,  making  it  possible  for 
them  to  identify  occupation-specific  skills.  Second,  occupation-specific  skills  are  identified  in 
terms  of  basic  and  cross-functional  skills,  making  it  possible  to  link  occupation-specific  skills  to 
a broader  cross-occupation  structure.  Third,  because  occupation-specific  skills  are  viewed  as 
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occupation-specific  instances  of  broader  skills,  the  tasks  assigned  to  an  occupation-specific  skill 
can  be  used  in  conjunction  with  task  survey  data  to  obtain  scale  scores  reflecting  the  importance, 
time  spent,  learning  difficulty,  and  performance  of  each  skill. 

These  procedures,  however,  are  based  on  three  key  assumptions.  The  first  major 
assumption  is  that  tasks  can  be  organized  into  subsets  based  on  the  kinds  of  activities  called  for. 
Thus,  one  might  group  all  installation  tasks  together.  Within  this  subset  of  tasks,  however,  there 
will  be  a set  of  lower-order  groupings  or  task  groups  which  represent  unique  occupation-specific 
procedures.  It  also  is  assumed  that  these  lower-order  task  groupings  will  not  only  share  certain 
procedures  but  they  will  also  share  common  knowledge  requirements. 

In  the  following  sections,  we  describe  the  procedures  introduced  in  Figure  16-7  in  some 
detail.  We  begin  with  the  second  step  inasmuch  as  the  procedures  for  identifying  tasks  have 
already  been  described  in  the  previous  section  examining  procedures  for  identifying  job  tasks. 
Organize  Tasks 

To  generate  task  sets  organized  by  a common  basic  or  cross-functional  skill  requirement, 
an  occupational  analyst  would  be  asked  to  link  the  tasks  to  basic  cross-functional  categories.  For 
example,  the  task  of  installing  heavy  outdoor  wiring  would  be  assigned  to  the  Installation  cross- 
functional skill.  This  sorting  of  tasks  into  basic  and  cross-functional  skill  categories  need  not  be 
solely  based  on  judgment.  In  fact,  research  currently  being  performed  by  the  first  three  authors  of 
this  chapter  intended  to  identify  verbs  commonly  associated  with  basic  and  cross-functional 
skills  can  be  used  to  guide  this  sorting  process. 

The  output  of  this  sorting  operation  would  be  several  sets  of  tasks.  Each  set  would 
contain  7 to  30  tasks,  all  of  which  shared  a cross-functional  or  basic  skill  requirement.  Based 
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upon  our  experience,  roughly  five  to  15  task  sets  will  be  required  to  adequately  cover  the  tasks 

involved  in  a given  job.  , . 

Identify  Occupation-Specific  Skills  and  Knowledges 

Once  tasks  have  been  sorted  into  the  basic  and  cross-functional  skills,  they  provide  a 
basis  for  identifying  the  occupation-specific  skills  and  knowledges.  The  results  of  the  task  sort 
are  used  to  construct  the  stimulus  material  needed  to  identify  occupation-specific  skills.  Figure 
16-8  illustrates  the  rating  sheets  used  in  this  exercise. 

In  this  exercise,  incumbents  or  supervisors  are  asked  to  review  the  tasks  subsumed  under 
a given  basic  or  cross-functional  skill  (e.g.,  the  27  tasks  shown  in  Figure  16-8  are  grouped  under 
the  cross-functional  skill  of  Installation).  As  they  review  this  task  list,  they  are  asked  to  identify 
groups  of  tasks  where  Teaming  one  task  would  help  you  learn  the  other  tasks  included  in  the 
group."  Each  such  group  of  tasks  defines  an  occupation-specific  skill.  Thus,  occupation-specific 
skills  are  defined  in  terms  of  transfer  and  associated  learning  requirements.  After  identifying 
those  groups  of  tasks  having  common  learning  requirements,  the  subject  matter  experts  are  asked 
to  assign  labels  to  each  of  the  groups  and  to  rate  the  difficulty  of  learning  each  task  in  the  group. 

Typically,  in  conducting  this  skill  generation  exercise,  it  is  necessary  to  use  incumbents  or 
supervisors  because  they  have  adequate  familiarity  with  the  job  and  pattern  of  skill  development 
on  the  job.  Moreover,  because  this  exercise  requires  some  breadth  of  understanding,  and  a 
relatively  high  degree  of  abstract  thinking,  supervisors,  as  opposed  to  incumbents,  appear  to 
provide  the  most  appropriate  group  for  acquiring  this  information. 

This  information  may  be  collected  either  in  subject  matter  expert  meetings,  or  through 
computer  assisted,  individualized  administration  of  the  skill  generation  exercise.  When  the  panel 
meeting  procedure  is  used,  it  is  generally  desirable  to  administer  this  exercise  to  two  panels,  each 
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containing  four  to  five  subject  matter  experts,  with  panel  members  being  asked  to  review  the 
occupation-specific  skills  proposed  by  other  panel  members  in  a “round-robin"  review  conducted 
after  initial  generation  of  the  occupation-specific  skills.  When  skills  are  being  generated  by 
individual  subject  matter  experts,  it  is  desirable  to  obtain  responses  from  a somewhat  larger  pool 
of  subject  matter  experts  to  compensate  for  the  lack  of  peer  review  and  feedback. 

Earlier,  we  noted  that  skills,  the  procedural  component  of  performance,  are  associated 
with  knowledges,  the  declarative  component.  Thus,  one  way  that  occupation-specific  knowledges 
might  be  identified  is  to  capitalize  on  this  implicit  association,  using  skills  as  a basis  for 
identifying  knowledges.  This  might  be  accomplished  by  asking  supervisors  to  indicate  the 
concepts  or  principles  required  to  be  able  to  master  the  tasks  subsumed  under  a given  occupation- 
specific  skill,  as  they  defined  it.  Alternatively,  they  might  be  asked  to  indicate  the  things  one 
needs  to  know  to  perform  the  general  kinds  of  activities  reflected  in  a given  skill.  Typically, 
subject  matter  experts  have  simply  been  asked  to  list  the  knowledges  associated  with  a certain 
occupation-specific  skill,  as  shown  in  the  bottom  part  of  Figure  16-8,  based  on  the  proposition 
that  many  knowledges  will  extend  across  the  tasks  included  in  a given  skill.  However,  at  times  it 
may  prove  useful  to  have  subject  matter  experts  indicate  those  individual  tasks,  from  among 
those  involved  in  the  occupation-specific  skill,  where  this  knowledge  is  viewed  as  essential  to 
good  performance. 

This  procedure  for  identifying  occupation-specific  knowledges,  like  the  skill 
identification  procedure,  will  generate  a number  of  potential  knowledge  labels.  Thus,  in 
identifying  occupation-specific  knowledges  as  well  as  occupation-specific  skills,  it  is  generally 
desirable  to  have  subject  matter  experts  review  and  edit  the  knowledges  being  proposed. 
Generally,  however,  it  will  also  be  necessary  for  an  occupational  analyst  to  review  and  edit  the 


Occupation-Specific  16-27 


resulting  list  of  occupation-specific  knowledges  and  skills  to  remove  redundancies.  This  review 
process  should,  of  course,  be  organized  in  terms  of  the  basic  and  cross-functional  skills  that 
provided  a framework  for  initial  identification  of  the  occupation-specific  skills  and  knowledges 

Field  Test  Results 

The  availability  and  validity  of  the  descriptive  information  obtained  using  these 
procedures  was  assessed  in  the  telecommunications  occupations  within  a large  organization 
(Mumford  & Supinski,  1995).  The  first  occupation,  telecommunications  technician,  involved  the 
installation,  repair,  and  maintenance  of  telecommunications  equipment.  The  second  occupation, 
telecommunications  systems  analyst,  involved  the  operation  and  control  of  automated 
telecommunications  systems.  Occupation-specific  skills  and  knowledges  were  identified  in  a set 
of  two  independent  panel  meetings,  conducted  using  supervisors  on  each  of  the  targeted 
occupations.  Prior  to  conducting  these  meetings,  104  analysts  and  437  technicians  had  been 
asked  to  rate  the  frequency,  importance,  learning  difficulty,  and  their  own  performance  with 
respect  to  each  of  the  tasks  occurring  on  their  job. 

Some  initial  evidence  for  the  reliability  of  the  descriptive  information  resulting  from 
these  procedures  was  obtained  by  contrasting  the  skills  identified  in  the  two  panels.  Here,  an 
analyst  developed  a consolidated  list  of  skills  and  then  identified  equivalent  occupation-specific 
skills  proposed  by  at  least  three  members  of  both  groups.  Overall,  86  occupation-specific  skills 
were  identified  for  technicians,  while  103  occupation-specific  skills  were  identified  for  the 
analysts.  Of  these  skills,  57%  were  proposed  by  members  of  both  technician  panels,  while  63% 
were  proposed  by  members  of  both  analyst  panels.  Similar  results  were  obtained  for  the 
occupation-specific  knowledges.  Here,  60%  of  the  technician  knowledges  and  59%  of  the  analyst 
knowledges  proposed  by  members  of  one  panel  were  proposed  by  members  of  the  other  panel. 
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Another  way  one  might  look  at  the  reliability  pf  the  descriptive  information  provided  by 
these  procedures  is  to  assess  whether  tasks  linked  to  a skill  share  something  in  common.  To 
address  this  issue,  an  occupational  analyst  reviewed  assignments  of  tasks  to  the  occupation- 
specific  skills  in  both  panels  and,  based  on  this  information,  assigned  tasks  to  the  relevant 
occupation-specific  skills.  Figure  16-9  presents  the  tasks  assigned  to  the  technicians’  physical 
troubleshooting  skill  and  the  analysts’  technology  planning  skill.  Subsequently,  these  tasks  were 
used  to  fomrskill  scales  by  aggregating  the  frequency,  importance,  learning  difficulty,  or 
performance  scores  obtained  for  each  task  incorporated  under  a skill.  Interrater  agreement 
coefficients  were  then  obtained  for  skill  scales  on  each  of  these  descriptors  by  drawing  20  judges 
at  random  from  the  pools  of  analysts  and  technicians  who  completed  the  task  survey.  In  the  case 
of  technicians,  it  was  found  that  the  median  interrater  agreement  coefficient  was  .78,  while  the 
median  interrater  agreement  coefficient  obtained  for  the  analysts  was  .82. 

Taken  as  a whole,  the  evidence  obtained  in  this  study  indicates  that  the  procedures 
described  above  allow  for  reasonably  accurate,  reliable  descriptions  of  the  occupation-specific 
skills  and  knowledges  required  on  a job.  Apparently,  independent  panels  produce  similar  lists  of 
skills  and  knowledges.  Further,  at  least  in  the  case  of  skills,  where  direct  linkages  to  relevant 
tasks  could  be  identified,  it  was  found  that  these  skills  reflected  tasks  that  displayed  substantial 
common  variance.  Nonetheless,  it  should  be  recognized  that  these  findings  provide  relatively 
little  evidence  bearing  on  the  meaningfulness  of  the  resulting  descriptive  information. 

Figure  16-10  presents  some  illustrative  skills  identified  for  the  technician  and  analyst 
jobs,  while  Figure  16-11  presents  some  illustrative  knowledges  involved  in  each  job.  As  may  be 
seen,  the  nature  of  these  skills  and  knowledges  is,  generally  speaking,  consistent  with  the  known 
characteristics  of  these  occupations.  For  example,  both  occupations  were  found  to  involve 
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technical  writing.  Hardware  repair  and  media  repair,  however,  were  unique  to  the  technician  job, 
while  system  parameter  programming  and  diagnostic  programming  were  unique  to  the  analyst 
job.  Thus,  the  nature  pf  the  similarities  and  differences  observed  among  the  occupation-specific 
skills  and  knowledges  obtained  on  these  jobs  provides  some  initial  evidence  for  those 
procedures. 

Somewhat  more  compelling  evidence  for  the  meaningfulness  of  the  descriptive 
information  obtained  using  those  procedures  was  provided  by  the  additional  analyses.  In  the  first 
set  of  analyses,  managers  were  asked  to  identify  incumbents  working  in  growth  positions  that 
involved  the  use  of  new  technologies.  The  scores  of  incumbents  in  emerging  jobs  were  compared 
to  the  scores  of  incumbents  in  more  traditional  positions  using  the  skill  scales  developed  by 
aggregating  performance  ratings  of  the  tasks  assigned  to  the  relevant  skills.  Emerging  technician 
positions  apparently  required  better  performance  with  respect  to  media  installation,  software  fault 
isolation,  system  fault  isolation,  and  test  equipment  application-findings  consistent  with  the  use 
of  more  complex  "black  box"  technology  in  the  emerging  technician  positions.  In  the  case  of 
systems  analysts,  similar  technical  skills  were  required  in  both  emerging  and  diminishing 
positions.  However,  perhaps  due  to  increased  sophistication  of  telecommunications  systems, 
emerging  positions  required  better  problem-solving  skills,  such  as  gathering  information  about 
technology,  generating  system  ideas,  evaluating  system  ideas,  status  monitoring,  and  feedback 
monitoring. 

Another  way  one  might  look  for  evidence  about  the  usefulness  of  these  procedures  is  by 
contrasting  the  profile  of  ratings  obtained  for  a certain  type  of  occupation-specific  skill  (e.g., 
problem  solving  skills)  across  the  emerging  and  diminishing  positions.  Figure  16-12  presents  the 
profile  correlations  obtained  for  the  emerging  and  diminishing  positions  within  the  occupations 
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of  technicians  and  analysts  with  respect  to  the  indices  of  importance,  time  spent,  performance, 

. and  learning  difficulty  derived  from  ratings  of  the  tasks  assigned  to  a given  occupation-specific 
skill.  Among  technicians,  the  importance,  time  spent,  and  learning  difficulty  of  skills  in  the 
emerging  and  traditional  positions  were  quite  similar.  What  differed  was  the  performance  profile 
for  technical,  basic,  and  social  skills.  Among  analysts,  the  most  clear-cut  finding  was  that 
different,  negatively  related  profiles  of  problem  solving  skills  were  required  in  the  two  types  of 
positions— a result  attributable  to  the  emphasis  on  abstract  thinking  required  by  new 
telecommunications  control  systems. 

These  analyses  show  how  data  gathered  using  these  procedures  can  be  applied  to  identify 
changing  job  requirements.  This  information  might,  of  course,  prove  of  some  value  in  both 
reengineering  and  retraining  efforts.  For  example,  technicians  may  need  better  performance  on 
"black  box"  technology  to  cope  with  emerging  changes,  while  analysts  may  need  to  acquire  a 
different  type  of  more  abstract,  deductive,  problem  solving  skills.  More  centrally,  however,  these 
findings  indicate  that  the  procedures  used  to  identify  occupation-specific  skills  and  knowledges 
are  capable  of  adequately  capturing  known  changes  in  job  demands— a finding  pointing  to  the 
construct  validity  of  the  resulting  information. 

Although  the  findings  obtained  in  this  initial  field  test  argue  for  the  reliability  and  validity 
of  the  descriptive  information  obtained  using  these  procedures,  three  points  should  be  borne  in 
mind.  First,  the  procedures  identify  the  content  or  nature  of  occupation-specific  skills  and 
knowledges,  but  they  tell  us  relatively  little  about  the  structure  and  strategies  involved  in  job 
performance.  Thus,  this  descriptive  information  may  need  to  be  supplemented,  using  techniques 
such  as  expert  review  comparisons  or  think-aloud  protocols,  if  there  is  a need  to  identify  the 
processes  shaping  performance.  Second,  these  procedures  are,  of  course,  not  the  only  approach 
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that  might  be  used  to  identify  occupation-specific  skills  and  knowledges.  For  example,  in  one 
variation  on  this  approach,  computer-assisted  expert  system  tutorials  are  employed  rather  than 
panel  meetings.  Alternatively,  knowledges  might  be  identified  using  the  general  cross-occupation 
taxonomy  presented  earlier,  rather  than  basing  knowledge  definition  on  the  associated  skills. 
Third,  this  is  an  initial  attempt  on  only  two,  related  jobs  that  both  require  relatively  well-educated 
workers.  No  doubt  these  procedures  may  evolve  and  almost  certainly  will  need  to  be  modified  for 
occupations  having  less  well-educated  workers.  Nonetheless,  the  procedures  sketched  out  above 
do  apparently  allow  one  to  initially  define  occupation-specific  skills  and  knowledges  within  the 
common  language  organizing  framework  provided  by  the  cross-occupation  descriptors— an 
important,  perhaps  essential,  characteristic  for  a viable  occupational  information  system. 

Tools 

Up  to  this  point,  we  have  focused  on  the  types  of  procedures  that  might  be  used  to 
identify  job  tasks  and  the  occupation-specific  skills  and  knowledges  needed  to  perform  job  tasks. 
However,  these  are  not  the  only  kinds  pf  occupation-specific  information  that  have  been  or  might 
be  used  to  describe  people’s  jobs.  Another  kind  of  occupation-specific  information  commonly 
used  to  describe  people's  jobs  are  the  tools  used  in  performance  tasks.  The  term  tools,  although 
apparently  unambiguous,  has  often  been  defined  in  a different  way.  Sometimes  tools  refers  solely 
to  physical  equipment,  while  at  other  times  this  category  is  extended  to  include  any  machines, 
equipment,  software,  or  job  aids  used  to  perform  job  tasks. 

In  developing  the  0*NET  occupational  information  system,  we  used  the  more 
encompassing  definition  of  tools.  More  specifically,  tools  were  viewed  as  any  hand  tool, 
machine,  piece  of  equipment,  software,  or  job  aid  that  was  used  in  performing  requisite  job  tasks. 
Thus,  a word  processing  computer  program  would  be  viewed  as  a tool  used  by  secretaries,  just  as 

1378 


0 


16-32  Mumford,  Sager,  Baughman,  & Childs 


a blow  torch  would  be  a tool  used  by  a welder.  This  broader  definition  was  applied  primarily 
because  it  provides  a more  comprehensive  definition  of  the  things  needed  to  perform  job  tasks, 
while  incorporating  recent  changes  in  technology  and  covering  a broader  range  of  jobs. 

It  is  not  at  all  difficult  to  identify  the  hand  tools,  equipment,  machines,  software,  and  job 
aids  that  are  used  in  a given  occupation.  Observation,  records,  and  documentary  material  can  all 
be  used.  A simpler  and  more  straightforward  strategy,  however,  is  simply  to  ask  incumbents  or 
supervisors  to  list  the  tools,  equipment,  machines,  software,  and  job  aids  used  regularly— more 
than  once  a month— on  their  jobs.  Although  this  listing  strategy  cannot  describe  exactly  how  a 
given  tool  is  applied,  which  requires  observation  and/or  a review  of  relevant  manufacturer’s 
specifications,  it  does  permit  reliable  identification  of  the  tools  used  on  the  job. 

The  problem  with  identifying  tools  as  so  broadly  defined,  however,  is  that  it  may  prove 
difficult  to  tie  tools  to  a broader  set  of  cross-occupation  variables.  Without  some  linkage  back  to 
the  broader  cross-occupation  variables,  it  becomes  difficult  to  include  information  about  tools  in 
a comprehensive  occupational  information  system.  Further,  unorganized  lists  of  tools  make  it 
difficult  to  determine  the  nature  and  conditions  of  tool  use.  Thus,  some  framework  is  required  for 
identifying  tools. 

Three  plausible  procedures  are  available  for  gathering  information  about  tools,  all  based 
on  the  premise  that  tools  should  be  identified  and  linked  to  specific  variables  used  to  describe 
people's  jobs.  Thus,  one  might  have  incumbents  list  the  tools  required  (a)  to  perform  particular 
job  tasks,  (b)  to  accomplish  the  tasks  subsumed  under  a more  general  work  activity  or  (c)  to 
execute  a particular  occupation-specific  skill.  Generally,  it  is  too  time  consuming  for  incumbents 
to  list  all  tools  associated  with  each  task  involved  in  a job.  Thus,  tool  lists  based  on  generalized 
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work  activities  or  occupation-specific  skills  appear  to  provide  a more  appropriate  strategy  for 
collecting  this  descriptive  information. 

As  part  of  a current  project  by  the  first  three  authors  of  this  chapter,  the  feasibility  of 
generating  tool  lists  using  occupation-specific  skills— and,  thus,  derivatively  broader  basic,  and 
cross-functional  skills— as  an  organizing  structure  are  being  examined.  Here,  after  generating 
occupation-specific  skills,  as  described  in  the  section  above,  incumbents  working  on  electrical 
engineering  jobs  were  asked  to  review  the  tasks  performed  when  applying  a skill.  They  were  then 
asked  to  list  the  tools  (i.e.,  equipment,  machines,  software,  and  job  aids)  involved  in  skilled 
performance  on  the  set  of  tasks.  In  a prior  review,  it  was  found  that  this  straightforward 
procedure  resulted  in  a comprehensive  list  of  tools  proving  particularly  useful  in  identifying 
software  and  job  aids  involved  in  skilled  performance.  Further,  by  linking  tools  to  skills,  the 
resulting  information  could  be  readily  used  in  designing  skill-based  training  programs. 

Although  the  results  obtained  in  this  initial  effort  argue  for  the  utility  of  identifying  and 
organizing  tools  in  terms  of  occupation-specific  skills,  in  some  cases  it  may  prove  more 
appropriate  to  generate  and  organize  tool  lists  based  on  generalized  work  activities.  Listing  tools 
required  to  perform  the  tasks  subsumed  under  a generalized  work  activity  places  less  burden  on 
subject  matter  experts,  requiring  the  development  of  fewer  lists.  By  the  same  token,  however, 
this  procedure  makes  it  more  difficult  to  link  tools  to  specific  job  content  and  may  result  in  a less 
comprehensive  definition  of  software  and  job  aids  involved  in  executing  relevant  job  tasks. 

Conclusions 

In  this  chapter  we  described  a number  of  methods  for  collecting  0*NET  occupation- 
specific  information.  The  first  was  a modified  version  of  the  Available  Task  Inventories 
approach.  This  approach  consists  of  using  task  lists  from  the  DOT  to  generate  occupation- 
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specific  task  rating  questionnaires  for  the  80  occupations  in  the  0*NET  prototype.  This  is 
considered  to  be  a good  interim  solution  for  including  occupation-specific  information  in  the 
prototype.  These  broad  task  statements  provide  a relatively  fast,  low  cost  mechanism  for 
collecting  occupation-specific  information.  Further,  they  provide  an  explicit  linkage  between  the 
new  0*NET  and  the  old  DOT. 

However,  there  are  a number  of  reasons  why  this  solution  is  less  attractive  for  the 
collection  of  task  data  once  development  moves  beyond  the  prototype  stage.  First,  our  modified 
version  of  the  approach  creates  a list  of  only  7 to  30  tasks  per  occupation.  A per  occupation  task 
list  of  this  size  does  not  generate  a sufficient  number  of  tasks  at  a sufficient  level  of  detail  to 
support  many  applications,  such  as  training  program  design,  nor  does  it  provide  an  adequate  basis 
for  identifying  occupation-specific  skills  and  knowledges.  Additionally,  this  modified  version 
suffers  from  all  of  the  disadvantages  inherent  in  the  fully  elaborated  Available  Task  Inventories 
approach.  For  example,  the  approach  is  completely  dependent  on  the  availability  and  quality  of 
existing  job  analyses.  Some  of  the  task  lists  from  these  analyses  already  are  out-of-date,  some 
were  developed  for  specific  purposes  resulting  in  task  lists  that  only  focus  on  some  parts  of  the 
occupation,  and  the  lists  are  likely  to  vary  greatly  in  quality  and  comprehensiveness. 

Because  of  these  disadvantages,  we  recommend  that  after  the  prototype  data  collection, 
the  GWA  approach  should  be  used  to  collect  0*NET  occupation-specific  information.  This 
approach  certainly  does  not  represent  the  only  set  of  procedures  that  might  be  used  to  generate 
task  information.  For  example,  in  many  cases,  some  combination  of  the  generalized  work 
activities  approach  and  the  action/object  linking  approach  appears  promising,  at  least  for  jobs 
where  the  work  force  is  well  educated  and  used  to  working  with  software  or  other  types  of 
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decision  aids.  Nonetheless,  the  GWA  approach  or  some  variation  of  it  does  have  some  attractive 
features  vis-a-vis  the  0*NET. 

By  explicitly  linking  the  generation  of  these  tasks  to  broader  cross-occupation 
taxonomies,  as  reflected  in  the  GW  As,  the  proposed  procedures  should  serve  to  facilitate 
organization  of  relevant  occupation-specific  information  in  terms  of  a broader,  more 
comprehensive  taxonomic  structure.  Further,  there  is  some  reason  to  suspect  that  the  kind  of 
hierarchically  structured  job  analysis  procedures  described  here  will  serve  to  ensure  more  rapid 
and  cost-effective  collection  of  occupation-specific  descriptive  information. 

The  availability  of  a cross-occupation  organizing  framework  is  of  some  importance,  not 
only  in  the  identification  of  job  tasks,  but  also  in  the  identification  of  occupation-specific  skills 
and  knowledges,  as  well  as  the  definition  of  requisite  tools.  With  regard  to  tools,  there  is,  of 
course,  little  problem  in  identifying  the  relevant  descriptors.  This  information  has  traditionally 
proven  difficult  to  use  because  it  could  not  be  organized  in  terms  of,  and  integrated  with,  other 
kinds  of  job  descriptors,  particularly  cross-occupation  descriptors.  The  procedures  presented  in 
this  chapter,  however,  show  how  those  kinds  of  occupation-specific  descriptors  can  be  identified 
and  organized  in  terms  of  the  cross-occupation  variables,  thereby  facilitating  use  of  this 
descriptive  data  in  a comprehensive  occupational  information  system. 

By  collecting  and  organizing  information  about  tools  within  a broader  framework,  it 
becomes  possible  to  apply  this  kind  of  descriptive  information  in  new  ways.  For  example,  one 
might  ask  the  question,  “What  are  the  tools  most  commonly  used  when  installing  electrical 
equipment  across  all  jobs  or  within  a family  of  jobs?”  This  information  is  clearly  of  some  real 
value  to  trainers  and  educators. 
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The  methods  sketched  out  above  also  appear  capable  of  addressing  a number  of  practical 
problems.  For  example,  it  is  expected  that  skills  boards  will  seek  to  identify  the  occupation- 
specific  skills  and  knowledges  required  in  various  occupations.  Not  only  do  the  procedures 
sketched  out  above  provide  a systematic  framework  for  defining  these  skills  and  knowledges,  but 
they  also  explicitly  link  skill  requirements  to  the  tasks  to  be  performed  in  the  occupation.  This 
should  facilitate  the  work  of  the  skills  boards,  while  providing  a relatively  economical  method 
for  the  identification  of  occupation-specific  skills  that  possess  some  reliability  and  validity. 

The  skills  boards,  however,  do  not  represent  the  only  area  where  occupation-specific 
information  is  needed.  In  training,  for  example,  there  often  is  a need  for  the  kind  of  occupation- 
specific  information  provided  by  those  procedures.  Trainers  need  to  know  what  occupation- 
specific  skills  and  knowledges  must  be  developed.  Further,  these  skills  and  knowledges  must  be 
developed  within  the  context  of  requisite  tasks.  The  procedures  described  above  would,  of 
course,  provide  trainers  with  this  background  information,  thereby  contributing  to  the  design, 
delivery,  and  evaluation  of  training  courses. 

In  addition  to  these  applications  of  occupation-specific  information,  the  type  of 
information  provided  by  these  procedures  might  be  used  to  address  a number  of  other  issues. 
First,  this  kind  of  occupation-specific  information  might  be  used  to  provide  guidelines  for  person 
assessment.  Second,  it  might  be  used  in  job  redesign  efforts.  Third,  it  might  prove  useful  in 
designing  wage  and  compensation  systems  based  on  skill  requirements. 

Certainly,  we  have  not  described  all  of  the  potential  applications  of  this  kind  of 
occupation-specific  information.  A number  of  other  applications  might,  of  course,  be  envisioned. 
Further,  in  many  cases,  effective  application  of  these  various  kinds  of  occupation-specific 
descriptors  may  require  the  collection  of  additional,  more  detailed  types  of  information.  Still, 
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development  programs,  for  example,  may  need  to  collect  information  about  the  processes  and 
procedures  involved  in  skill  application  through  techniques  such  as  error  analysis,  think-aloud 
protocols,  or  expert-novice  comparisons.  It  should  be  recognized  that  many  of  these  procedures, 
although  promising,  are  still  under  development.  Thus,  provisions  and  extensions  of  those 
procedures  can  be  expected  as  further  work  progresses.  Even  bearing  this  caveat  in  mind, 
however,  we  believe  that  extension  of  the  0*NET  cross-occupation  framework  to  the 
identification  of  occupation-specific  descriptors  makes  an  important  point.  There  is  no  inherent 
conflict  between  the  cross-occupation  approach  employed  in  the  0*NET  content  model  and  the 
more  traditional  occupation-specific  approach  commonly  used  in  job  analysis.  Instead,  the  cross- 
occupation  structure  may  facilitate  the  identification  and  organization  of  more  specific  types  of 
descriptive  information,  thereby  making  it  possible  to  obtain  more  comprehensive,  up-to-date 
information  which  can  be  used  more  efficiently  to  address  a number  of  new  concerns. 
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Appendix  16- A 

A Method  of  Identifying  Occupation-Specific  Tasks.  Skills.  Knowledges,  and  Tools 

As  part  of  a project  for  a large  U.S.  Department  of  Defense  agency,  the  American 
Institutes  for  Research  is  developing,  testing,  and  implementing  a computer-based  method  of 
collecting  occupation-specific  information  that  is  structured  by  the  0*NET’s  content  model.  The 
method  relies  on  five  computer  modules  that  incumbents  and  occupational  analysts  use  to 
generate  and  consolidate  occupation-specific  information.  Below  is  a brief  description  of  each 
computer  module. 

1.  Task  Generation 

Participants:  Twenty-four  incumbents  representing  a particular  occupation. 

Activity;  This  computer  module  helps  incumbents  develop  individual  lists  of  tasks 
that  describe  the  occupation.  The  program  starts  by  providing  a brief 
tutorial  that  describes  the  structure  of  task  statements,  explaining  that  the 
essence  of  a task  is  a verb  and  an  object  (e.g.,  “load  boxes”),  and  that  task 
statements  can  be  made  more  specific  by  adding  object  modifiers  and 
statement  qualifiers.  For  example,  “Load  wooden  boxes  onto  trucks.” 

Next,  the  module  presents  each  incumbent  with  the  titles  and  definitions  of 
0*NET’s  Generalized  Work  Activities  (GW As).  The  incumbent  must 
select  between  6 and  12  GW  As  that  represent  the  majority  of  work 
activities  carried  out  on  their  job.  The  module  then  helps  the  incumbent 
generate  tasks  for  each  of  the  selected  GW  As,  one  GWA  at  a time.  This 
task  generation  process  starts  by  presenting  the  incumbent  with  brief  lists 
of  verbs  and  objects  that  are  commonly  associated  with  the  selected  GWA 
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Outcome: 

2.  Task  Editing 

Participants: 

Activities: 


Outcome: 


and  allows  the  incumbent  to  (1)  create  task  statements  using  pairs  of 
existing  verbs  and  objects,  (2)  create  and  pair  their  own  verbs  and  objects, 
and  (3)  develop  complete  task  statements  for  that  GWA  by  adding  object 
modifiers  and  statement  qualifiers  as  necessary  to  the  verb/object  pairs. 
This  process  is  repeated  until  the  incumbent  has  generated  tasks  for  each 
of  the  selected  GW  As. 

The  output  of  this  computer  module,  administered  to  approximately  24 
incumbents,  is  a list  of  occupation-specific  task  statements  for  each 
incumbent  in  a particular  occupation. 

An  occupational  analyst  and  a content  specialist.  A content  specialist  is  an 
individual  who  has  considerable  awareness  of  the  industry  and  family  of 
occupations  related  to  the  target  occupation. 

This  computer  module  helps  the  occupational  analyst  and  the  content 
specialist  (i.e.,  the  “job  analysis  pair”)  edit  the  task  lists  generated  by  the 
approximately  24  incumbents  into  a consolidated  list  of  occupation- 
specific  tasks.  Basically,  the  job  analysis  pair  uses  the  module  to  carry  out 
such  editing  as  choices  eliminating  inappropriate  and  redundant  tasks  and 
clarifying  language. 

The  output  of  this  computer  module  is  a list  of  occupation-specific  tasks 
that  are  used  in  subsequent  modules  to  generate  other  occupation-specific 
information.  This  list  of  tasks  is  also  used  directly  to  develop  an 
occupation-specific  task  survey  where  incumbents  rate  tasks’  (1)  relevance 


to  the  occupation,  (2)  frequency  of  performance,  (3)  importance  to 
performance,  and  (4)  difficulty. 

3.  Task/Skill  Linkage  or  Task  Clarification 


Participants: 

The  job  analysis  pair  referred  to  in  the  Task  Editing  module  (i.e.,  an 
occupational  analyst  and  a content  specialist). 

Activity: 

This  computer  module  helps  the  job  analysis  pair  link  each  task  output 
from  the  task  editing  module  to  relevant  Basic  and  Cross-Functional  Skills 
from  the  0*NET  content  model  using  a simple  point  and  click  procedure. 
The  job  analysis  pair  is  presented  with  each  task  and  the  titles  and 
definitions  of  the  Basic  and  Cross-Functional  Skills.  They  then  link  each 
task  to  the  skills  that  are  judged  to  be  most  critical  to  performance  of  the 
task.  The  module  sets  an  upper  limit,  stipulating  that  no  more  than  five 
skills  can  be  linked  to  a task. 

Outcome: 

The  output  from  this  computer  module  is  a linkage  between  each  task  and 
relevant  Basic  and  Cross-Functional  Skills.  This  linkage  is  a required  input 
to  the  next  computer  module,  which  presents  incumbents  with  each 
relevant  Basic  and  Cross-Functional  Skill  and  its  associated  tasks. 

4.  Occupation-Specific  Skill,  Knowledge,  and  Tool  Identification 


Participants: 

The  same  24  incumbents  who  completed  the  first  computer  module. 

Activity: 

» 

The  primary  purpose  of  this  computer  module  is  to  help  each  incumbent 
independently  identify  occupation-specific  skills,  knowledges,  and  tools. 
Sequentially,  the  program  presents  the  title  and  definition  of  each  Basic 
and  Cross-Functional  Skill  that  was  linked  to  tasks  in  the  task/skill  linkage 
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module.  The  Basic  or  Cross-Functional  Skill  is  presented,  accompanied  by 
a list  of  all  the  tasks  that  were  linked  to  it.  First,  fhe  incumbent  is  asked  to 
rate  each  task  in  terms  of  how  long  it  takes  to  master  it,  on  a scale  that 
spans  from  0-3  months  to  more  than  8 years.  Next,  the  incumbent  is 
required  to  use  a point  and  click  procedure  to  organize  the  tasks  linked  to  a 
particular  Basic  or  Cross-Functional  Skill  into  smaller  groups  of  tasks.  The 
instructions  indicate  that  these  smaller  groups  of  tasks  should  be  tasks  that 
are  “learned  together.”  The  incumbent  is  required  to  provide  a label  for 
each  of  these  small  groups  of  tasks.  These  small  groups  of  tasks  and  their 
labels  are  used  in  the  subsequent  module  to  identify  occupation-specific 
skills.  Finally,  for  each  small  group  of  tasks,  the  incumbent  is  asked  to  list 
the  kinds  of  knowledge  and  tools  that  are  necessary  for  performing  these 
tasks. 

Outcome:  The  output  of  this  computer  module,  administered  to  approximately  24 

incumbents,  are  labeled  groups  of  tasks  for  each  incumbent  that  define 
preliminary  occupation-specific  skills,  with  the  knowledge  and  tools 
necessary  for  their  performance. 

5.  Occupation-Specific  Skill.  Knowledge,  and  Tool  Editing 

Participants:  The  job  analysis  pair  referred  to  originally  in  the  Task  Editing  module 

(i.e.,  an  occupational  analyst  and  a content  specialist). 

Activities:  This  computer  module  helps  the  job  analysis  pair  edit  the  information 

provided  by  the  approximately  24  incumbents  who  individually  completed 
the  Occupation-Specific  Skill,  Knowledge,  and  Tool  Identification 


module.  This  computer  module  analyzes  the  preliminary  occupation- 
specific  skills  generated  by  all  of  the  incumbents  and  suggests  a single 
integrated  list  of  occupation-specific  skills  for  the  occupation.  The  module 
helps  the  job  analysis  pair  (1)  work  with  and  edit  this  suggested  list  into  a 
single  list  of  occupation-specific  skills  for  the  occupation,  and  (2)  edit  the 
knowledges  and  tools  and  assign  them  to  appropriate  occupation-specific 
skills. 

Outcome:  The  output  of  this  computer  module  is  a final  list  of  occupation-specific 

skills  for  a particular  occupation.  Each  occupation-specific  skill  is 
accompanied  by  (1)  a title,  (2)  relevant  tasks,  and  (3)  relevant  knowledges 
and  tools. 

Summary 

This  appendix  is  a description  of  a computer-based  method  that  is  currently  being 
employed  for  identifying  occupation-specific  tasks,  skills,  knowledges,  and  tools.  Occupation- 
specific  information  is  an  important  part  of  the  0*NET  content  model;  however,  it  is  often  very 
expensive  and  time  consuming  to- collect.  The  method  proposed  here  was  designed  to  provide 
valid  occupation-specific  information  while  minimizing  the  time  and  financial  resources  required 
to  collect  it. 
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Figure  16-1 

Example  Occupation-Specific  Tasks  Questionnaire 


Instructions  for  Making  Task  Ratings 


In  this  questionnaire  you  will  be  presented  with  a list  of  tasks.  A task  is  an  action  or  set  of  actions  performed  together  to 
accomplish  an  objective.  This  list  of  tasks  will  be  specific  to  the  job  you  are  describing. 

For  each  task,  please  make  the  following  three  ratings:  RELEVANCE,  FREQUENCY,  and  IMPORTANCE. 

(1)  RELEVANCE.  If  the  task  is  NOT  RELEVANT  at  all  to  performance  on  the  job,  mark  through  the  "0"  in  the  NOT 
RELEVANT  column.  Carefully  read  the  task  before  deciding  whether  it  is  RELEVANT  or  NOT  RELEVANT  to  this  job. 
If  you  select  the  ”0"  in  the  NOT  RELEVANT  column,  however,  there  is  no  need  to  complete  the  IMPORTANCE  and 
FREQUENCY  ratings  described  below.  If  the  task  is  part  of  this  job,  rate  IMPORTANCE  and  FREQUENCY. 

(2)  FREQUENCY.  (Do  not  complete  if  NOT  RELEVANT  was  selected.)  Ask  yourself,  "How  often  is  this  task  performed 
on  this  job?  For  example.  Interact  with  potential  customers"  is  a task  that  an  employee  in  one  job  might  perform  only 
"once  per  week  or  less,"  but  an  employee  in  another  job  might  perform  "hourly  or  more  often." 

Rate  the  FREQUENCY  with  which  a task  is  performed  by  marking  through  the  appropriate  number,  from  1 (indicating 
that  the  task  is  performed  once  per  year  or  less  often)  to  7 (indicating  that  the  task  is  performed  hourly  or  more  often)  on 
the  FREQUENCY  scale. 

(3)  IMPORTANCE.  (Do  not  complete  if  NOT  RELEVANT  was  selected.)  Ask  yourself,  "How  important  is  this  task  to 
performance  on  this  job?"  For  example,  "Develop  objectives  and  strategies  to  guide  the  organization"  might  be  very 
important  for  an  employee  in  one  job,  but  less  important  for  another  job.  For  the  second  job,  however,  "Provide 
performance  feedback  to  subordinates"  might  be  very  important 

Rate  the  IMPORTANCE  of  the  task  for  performance  on  the  job  by  marking  through  the  appropriate  number,  from  1 
(indicating  that  the  task  is  of  no  importance)  to  5 (indicating  that  the  task  is  extremely  important)  on  the  IMPORTANCE 
scale. 

The  first  two  tasks  show  how  Frequency  and  Importance"  differ.  An  employee  in  a particular  job  indicates  that  "Land  a 
plane  under  emergency  conditions"  occurs  only  "once  per  year  or  less,”  but,  the  task  is  an  "extremely  important"  part  of 
the  employee's  job.  In  contrast,  the  employee  indicates  that  task  2 is  performed,  "several  times  per  day,"  but  is  less 
important  than  task  1.  Finally,  task  3 is  not  part  of  this  job,  so  the  employee  indicates  this  by  selecting  the  "Not  Relevant" 
circle. 


[three  completed  examples  were  included  here] 


Turn  the  page  to  begin  the  Tasks  Questionnaire. 
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Figure  16-1  (continued) 

Example  Occupation-Specific  Tasks  Questionnaire 


Task 


Frequency 


Importance 


1 Observes,  evaluates,  and 
records  patient  data. 

2 Reviews  diagnostic  tests. 

3 Conducts  laboratory  tests. 

4 Aids  physician  and  other  health 
care  professionals  during 
treatment  and  examination  of 
patients. 

5 Performs  physical  examinationsJI 
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6 Administers  injections,  ||  [|  @ 

medications  and  treatments. 

7 Cleans  and  sterilizes  ||  ||  (5) 

instruments  and  equipment. 

8 Consults  with  other  medical  ||  | @ 
professionals  on  policy  and 
patient  care. 

9 Supervises  and  coordinates  ||  ||  @ 
activities  of  nursing  personnel. 

10  Devetops  standards  and  p ® 

procedures  for  providing  nursing[| 
care. 

11  Provides  nursing  orientation,  ||||  © 
teaching,  and  guidance  to  staff,  j 

12  Identifies  problems  and  instructs))  [|  @ 
and  advises  personnel  in 
infection  control  procedures. 

1 3 Investigates  infection  control  ||  ||  © 
problems  and  follows  up  with 
persons  exposed  to  infection 
and  diseases. 

14  Administers  anesthetics.  1 j © 

15  Provides  instruction  in  health  1 1|  © 
education  and  disease 
prevention 
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Figure  16-1  (continued) 

Example  Occupation-Specific  Tasks  Questionnaire 


16  Provides  first  aid. 

17  Assesses  community  health 
care  needs. 

18  Provides  employee  health 
services  within  industrial 
organization. 

18  Plans  and  participates  in  school 
health  program. 

20  Participates  in  surveys  and 
research  studies. 

21  Counsels  and  provides  support 
for  patients  and  families. 


Additional  Relevant  Tasks 
Please  write  in  additional 
relevant  tasks  and  provide  a 
rating 
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Figure  16-2 

Cross-Walk  of  Occupation-Specific  Tasks  to  Generalized  Work  Activities 
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Figure  16-2  (continued) 

Cross-Walk  of  Occupation-Specific  Tasks  to  Generalized  Work  Activities 
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Figure  16-2  (continued) 

Cross-Walk  of  Occupation-Specific  Tasks  to  Generalized  Work  Activities 
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Figure  16-2  (continued) 

Cross-Walk  of  Occupation-Specific  Tasks  to  Generalized  Work  Activities 
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Figure  16-2  (continued) 

Cross-Walk  of  Occupation-Specific  Tasks  to  Generalized  Work  Activities 
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Figure  16-2  (continued) 

Cross-Walk  of  Occupation-Specific  Tasks  to  Generalized  Work  Activities 
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Figure  16-2  (continued) 

Cross-Walk  of  Occupation-Specific  Tasks  to  Generalized  Work  Activities 
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Figure  16-2  (continued) 

Cross-Walk  of  Occupation-Specific  Tasks  to  Generalized  Work  Activities 
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Figure  16-2  (continued) 

Cross-Walk  of  Occupation-Specific  Tasks  to  Generalized  Work  Activities 
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Figure  16-2  (continued) 

Cross-Walk  of  Occupation-Specific  Tasks  to  Generalized  Work  Activities 
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Figure  16-2  (continued) 

Cross-Walk  of  Occupation-Specific  Tasks  to  Generalized  Work  Activities 
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Figure  16-2  (continued) 

Cross-Walk  of  Occupation-Specific  Tasks  to  Generalized  Work  Activities 
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Figure  16-2  (continued) 

Cross-Walk  of  Occupation-Specific  Tasks  to  Generalized  Work  Activities 
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Figure  16-2  (continued) 

Cross-Walk  of  Occupation-Specific  Tasks  to  Generalized  Work  Activities 
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Figure  16-2  (continued) 

Cross-Walk  of  Occupation-Specific  Tasks  to  Generalized  Work  Activities 
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Figure  16-2  (continued) 

Cross-Walk  of  Occupation-Specific  Tasks  to  Generalized  Work  Activities 
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Figure  16-2  (continued) 

Cross-Walk  of  Occupation-Specific  Tasks  to  Generalized  Work  Activities 
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Figure  16-2  (continued) 

Cross-Walk  of  Occupation-Specific  Tasks  to  Generalized  Work  Activities 
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Figure  16-2  (continued) 

Cross-Walk  of  Occupation-Specific  Tasks  to  Generalized  Work  Activities 
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Figure  16-2  (continued) 

Cross-Walk  of  Occupation-Specific  Tasks  to  Generalized  Work  Activities 
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Figure  16-3 

Example  Instructions  for  the  Generation  of  Occupation-Specific  Skills  and  Knowledges 


SUBCATEGORIES  AND  TASKS 


Now  that  you  have  identified  broad  behavioral  categories  related  to  your  job,  the  next 
step  is  to  break  the  categories  into  subcategories,  or  more  specific  groups  of  activities. 
After  breaking  categories  into  subcategories,  each  subcategory  will  be  further  broken 
down  into  specific  components,  or  tasks. 

Let's  say  that  you  identified  "self  development"  as  a behavioral  category  related  to  your 
job.  The  next  step  would  be  to  describe  smaller  parts  of  this  category.  For  example, 
"participate  in  training,"  "keep  up  with  new  information,"  and  "self  assessment"  are  all 
subcategories  important  to  self  development.  In  this  way,  we  may  create  several 
subcategories  for  any  given  category. 

As  you  identify  each  subcategory,  please  list  all  of  the  specific  tasks  performed  on  your 
job  that  fall  under  that  subcategory.  A good  description  of  a "task"  includes  two  parts: 

1.  a specific  action  (what  you  do),  and 

2.  the  purpose  of  that  action  (why  you  do  it) 

EXAMPLE: 

The  subcategory  "Keeping  up  with  new  information"  might  include  tasks  such  as: 

1 . Subscribe  to  job  related  publications  to  obtain  new  articles. 

2.  Read  newspapers  to  look  for  job  related  information. 

3.  Attend  conferences  related  to  the  job. 

It  is  important  to  list  all  the  tasks  that  you  do,  no  matter  how  unimportant  you  think  they 
are.  The  purpose  of  this  "category  — > subcategory  — » task  process"  is  to  generate  more 
detailed  information  about  what  you  do  on  the  job.  After  you  have  listed  all  possible  tasks 
under  a subcategory,  move  on  to  identifying  the  next  subcategory. 


Note.  Adapted  from  Application  of  a Model  to  Develop  Occupation-Specific  Skills,  by 
M.  D.  Supinski  and  E.  Supinski,  1995,  Washington,  DC:  American  Institutes  for 
Research. 
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Figure  16-4 

Illustrative  Tasks  for  Commercial  Sales  in  the  Electrical  Products  Industry 


• Give  distributors  updated  pricing  information 

• Train  new  distributor  sales  representatives  about  applications  of  Firm  X products, 
promotional  activities,  and  new  products 

• Inform  distributor  of  the  pricing  terms  for  a particular  job 

• Fill  out  SPA  reports 

• Discuss  pricing  issues  with  distributors 

• Apply  knowledge  learned  in  the  past  to  solve  similar  or  related  problems 

• Negotiate  with  pricing  department  about  annual  contracts  and  SPA s 

• Go  on  joint  calls  with  distributor  representative  in  order  to  assess  user’s  lighting  needs  and 
recommend  appropriate  product  for  environment 

• Respond  to  requests  from  immediate  management  that  require  immediate  attention 

• Implement  local  incentive  plan  and  meet  with  distributors  to  explain  the  plan 

• Check  report  for  expiring  SPAs  to  determine  which  ones  to  renew 

• Receive  performance  appraisal  from  manager 

• Contact  distributor  management  in  order  to  convince  them  to  allocate  time  and  resources 
for  training  or  promotional  activities 

• Contact  manager  to  notify  him/her  of  a problem  that  cannot  be  immediately  resolved  or 
that  cannot  be  solved  by  the  sales  representative  alone 

• Talk  to  distributors  to  find  out  about  solicitation  by  competitors  and  prices 

• Go  over  annual  reviews  with  distributor  and  set  mutual  goals  for  the  next  year 

• Train  experienced  distributor  sales  representatives  about  applications  of  Firm  X products, 
promotional  activities,  and  new  products 


Figure  16-5 

Automated  Job  Analysis  System 
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Figure  16-6 

Example  Task  Statements  Generated  Using  Automated  Job  Analysis  System 

Judge  reports  from  skill-based  systems  on  computers. 

Survey  users  on  skill-based  systems. 

Survey  users  of  computerized  job  analysis  tools. 

Read  manuals  for  skill-based  systems. 

Read  reports  on  organizational  human  resources  initiatives. 

Develop  goals  for  pilot  projects. 

Record  procedures  that  are  used  to  pilot  test  systems. 

Describe  services  to  supervisors  and  customers. 

Adapt  established  procedures  to  adapt  to  new  situations. 

Adapt  strategies  to  the  project. 

Design  leadership  classes. 

Facilitate  executive-level  and  mentoring  classes  for  other  employees. 
Evaluate  programs  for  teacher  certification. 


Figure  16-7 

General  Procedures  for  Identifying  Occupation-Specific  Skills  and  Knowledges 


Identify  tasks 


o 


Organize  tasks  into 
basic  and  cross-functional 
skills 


o 


Group  tasks  within  basic 
and  cross-functional  skills 
into  occupation-specific 
skills  based  on  transfer 


o 


Identify  knowledges 
or  principles 
underlying  skill 
application 


Figure  16-8 

Example  Skill  and  Knowledge  Ratine  Exercise 


1 . Climbs  ladders  or  racks  to  install 
duct  work  or  run  cables 

2.  Conduct  de-installs 


3.  Configure  network  equipment 
based  on  cable  infrastructure. 

4.  Configure  software  for  work 
stations 

5.  Crawl  under  crawl  spaces  to 
install  cables 

6.  Install  duct  work 


7.  Install  sleepers 


8.  Install  cables  and  fibers 


9.  Install  circuits  (e.g.(  point  to 
point,  packet  switched, 
multiplexers) 

10.  Install  floor  tiles 


INSTALLATION 

11.  Install  network  infrastructure 
equipment  (e.g.  video,  voice, 
data) 

12.  Install  power 


13.  Install  racks 


14.  Install  services  (e.g.  opscom, 
tele-conferencing) 

15.  Make  cables  and  cable 
connectors 

16.  Pack  and  ship  equipment  and 
documents 

17.  Prepare  and  install  test 
equipment 

18.  Pull  cables 


19.  Reload  and  reconfigure 
software  in  repairing  personal 
computers 

20.  Remove  components  from 
computer  and  other  equipment. 


21 .  Run  conduit  for  installation 


22.  Sanitize  old  computers  and 
donfae  to  local  high  schools 

23.  Use  Agency,  DOD,  OSHA  and 
other  safety  guidelines 

24.  Use  hand  and  power  tools  (e.g., 
screw  driver,  hammer,  soldering 
iron) 

25.  Use  installation  SOPs  (e.g., 
complete  install  check  list) 

26.  Use  inter-operability  standards 
when  connecting  networks  with 
other  agencies. 

27.  Use  wiring  color  codes  to 
determine  appropriate  wiring 
configuration. 


YOUR  TASK  GROUPS  BASED  ON  COMMON  LEARNING  REQUIREMENTS 


Task  Group  One: 


By  Task  Number 
From  1 low  - 15  high 


By  Task  Number 
From  1 low  - 15  high 


By  Task  Number 
From  1 low  - 15  high 


YOUR  TASK  KNOWLEDGES 
Task  group  one  knowledges 

Task  group  two  knowledges 

Task  group  three  knowledges 


ERIC 


Note.  Adapted  from  Application  of  a Model  to  Develop  Occupation-Specific  Skills,  by  M.  D.  Supinski  and  E. 
Supinski,  1995,  Washington,  DC:  American  Institutes  for  Research. 


Figure  16-9 

Examples  of  Tasks  Assigned  to  Occupation-Specific  Skills 


Skill 

Tasks 

Physical  Troubleshooting 
(Telecommunications  Technician) 

Conduct  site  inspections 
Follow  troubleshooting  SOPs 

Identify  sources  of  transmission  systems  errors  and  faults 
Learn  interdependency  of  systems  to  increase  accuracy  of 
fault  isolations 

Listen  to  equipment  to  detect  mechanical  problems 
Make  necessary  adjustments  to  systems  based  on  analysis 
from  test  equipment 

Use  senses  of  smell  and  touch  to  determine  if  problem 
exists  (e.g.,  hot  cable) 

Technology  Planning 
(Telecommunications  Analyst) 

Create  circuit  installation  plans  (e.g.,  point  to  point,  packet 
switched,  multiplexers) 

Create  installation  plan  for  services  (e.g.,  opscom,  tele- 
conferencing) 

Decide  how  to  add  users  to  network 
Develop  architecture  plan  to  guide  technology  upgrades 
Develop  implementation  plan  for  new  technology 
Make  decisions  regarding  changes  or  additions  to 
technology  on  network 

Use  software  tools  to  model  network  topology 

Note.  Adapted  from  Application  of  a Model  to  Develop  Occupation-Specific  Skills,  by  M.  D. 
Supinski  and  E.  Supinski,  1995,  Washington,  DC:  American  Institutes  for  Research. 
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Figure  16-10 

Examples  of  Occupation-Specific  Skills  Identified  for  Two  Occupations 


Occupation 

Example  Occupation-Specific  Skills 

Telecommunications  Technician 

Technology  Reading 

Technical  Writing 

Customer  Awareness 

Presenting  Technical  Recommendations 

Customer  Technical  Support 

System  Fault  Isolation 

Hardware  Repair 

Media  Repair 

Architecture  Selection 

Program  Budgeting 

Generating  Technical  Ideas 

Operational  Review 

Telecommunications  Analyst 

Professional  Reading 
Technical  Writing 

Team  Performance  Monitoring  Technical  Support 

Test  Equipment  Application 

Evaluating  Test  Data 

System  Parameter  Programming 

Diagnostic  Programming 

Network  Operations 

Technical  Decision  Making 

Contracting 

Note,  Adapted  from  Application  of  a Model  to  Develop  Occupation-Snecifir.  Skills  by  M.  D. 
Supinski  and  E.  Supinski,  1995,  Washington,  DC:  American  Institutes  for  Research. 


Figure  16-11 

Examples  of  Occupation-Specific  Knowledges  Identified  for  Two  Occupations 


Occupation 

Example  Occupation-Specific  Knowledges 

Telecommunications  Technician 

Differences  Among  Phone  Systems 
Location  of  Circuits 
Architectural  Requirements 
Resource  Constraits 
Technical  Terminology 
Principles  of  Grounding 
Safety  Procedures 

Telecommunications  Analyst 

Technical  Terminology 
Data  Collection  Procedures 
Training  Principles 
Resource  Requirments 
Evaluation  Techniques 
Customer  Requirements 

Note.  Adapted  from  Application  of  a Model  to  Develop  Occupation-Specific  Skills,  by  M.  D. 
Supinski  and  E.  Supinski,  1995,  Washington,  DC:  American  Institutes  for  Research. 


Figure  16-12 

Profile  Comparisons  for  Emerging  and  Diminishing  Positions  Within  Two  Occupations 


Scale 

Importance  | Time  Spent  1 Performance 

Difficulty 

Telecommunications  Technician 

Overall 

r .69 

.67 

.21 

.77 

Basic  Skills 

.72 

.72 

.11 

.87 

Technical  Skills 

■ .48 

.64 

1 .18 

.85 

Problem  Solving  Skills 

.61 

.74 

.48 

.68 

Social  Skills 

.76 

.66 

.05 

.41 

Resource  Management  Skills 

.87 

.69 

.38 

.74 

Telecommunications  Analyst 

Overall 

.46 

.32 

.04  1 

.22 

Basic  Skills 

.68 

.14 

.32 

.60 

Technical  Skills 

.56 

.24 

.41 

.61 

Problem  Solving  Skills 

-.29 

-.24 

-.45 

.38 

Social  Skills 

.88 

.24 

.28 

.48  . 

Systems  Skills 

.14 

.89 

.26 

.83 

Resource  Management  Skills 

.14 

.29 

.01 

-.49 

Not^  Adapted  from  Application  of  a Model  to  Develop  Occupation-Specific  Skills1  by  M.  D. 
Supinski  and  E.  Supinski,  1995,  Washington,  DC:  American  Institutes  for  Research. 


Table  16-1 

Reliability  of  Task  Ratines  Considering  Varying  Numbers  of  Raters:  Occupation-Specific  Tasks 

Number  of  Raters  on  Each  Variable 


Frequency  Importance 


Occupation 

n 

Dca 

r,b 

E30c 

£k 

£1 

130 

15005  Education  Administrators 

39 

.97 

.42 

.96 

.95 

.34 

.94 

19005  General  Managers  & Top  Executives 

135 

.99 

.45 

.96 

.98 

.30 

.93 

25 105  Computer  Programmers 

18 

.87 

.27 

.92 

.86 

.25 

.91 

31305  Teachers,  Elementary  School 

79 

.99 

.46 

.96 

.98 

.43 

.96 

32502  Registered  Nurses 

93 

.99 

.47 

.96 

.98 

.40 

.95 

32902  Medical  & Clinical  Laboratory  Technologists 

7 

.90 

.58 

.98 

.89 

.53 

.97 

49008  Salespersons,  Except  Scientific  & Retail 

39 

.97 

.43 

.96 

.97 

.43 

.96 

49011  Salespersons,  Retail 

58 

.97 

.39 

.95 

.97 

.36 

.94 

49021  Stock  Clerks,  Sales  Floor 

24 

.76 

.12 

.80 

.81 

.15 

.84 

49023  Cashiers 

68 

.97 

.34 

.94 

.97 

.31 

.93 

51002  First  Line  Supervisors,  Clerical/Administrative 

165 

.99 

.32 

.93 

.98 

.27 

.92 

53102  Tellers 

17 

.95 

.55 

.97 

.94 

.49 

.97 

53905  Teachers'  Aides  & Assistants,  Clerical 

23 

.87 

.22 

.89 

.83 

.18 

.87 

55108  Secretaries,  Except  Legal  & Medical 

238 

1.00 

.51 

.97 

.99 

.32 

.94 

55305  Receptionists  & Information  Clerks 

29 

.98 

.58 

.98 

.97 

.56 

.97 

55338  Bookkeeping,  Accounting,  & Auditing  Clerks 

87 

.98 

.37 

.95 

.98 

.33 

.94 

55347  General  Office  Clerks 

237 

1.00 

.46 

.96 

.99 

.31 

.93 

61005  Police  & Detective  Supervisors 

38 

.96 

.39 

.95 

.93 

.25 

.91 

63014  Police  Patrol  Officers 

65 

.99 

.58 

.98 

.98 

.45 

.96 

65008  Waiters  & Waitresses 

53 

.98 

.43 

.96 

.97 

.37 

.95 

65026  Cooks,  Restaurant 

13 

.89 

.38 

.95 

.87 

.34 

.94 

65038  Food  Preparation  Workers 

69 

.97 

.30 

.93 

.96 

.27 

.92 

66008  Nursing  Aides,  Orderlies,  & Attendants 

49 

.97 

.44 

.96 

.97 

.40 

.95 

67005  Janitors  & Cleaners 

98 

.99 

.48 

.96 

.98 

.32 

.93 

85132  Maintenance  Repairers,  General  Utility 

100 

.96 

.20 

.88 

.96 

.21 

.89 

87902  Earth  Drillers,  Except  Oil  & Gas 

16 

.95 

.53 

.97 

.94 

.48 

.97 

92974  Packaging  & Filling  Machine  Operators 

31 

.77 

.10 

.76 

.78 

.10 

.77 

97102  Truck  Drivers,  Heavy  or  Tractor  Trailer 

36 

.97 

.44 

.96 

.96 

.42 

.96 

97111  Bus  Drivers,  Schools 

34 

.98 

.66 

.98 

.98 

.56 

.97 

Note.  Reliability  estimates  are  based  on  incumbents’  ratings  of  occupation-specific  tasks.  Decimals  are 
omitted. 

“Observed  reliability  estimates  were  obtained  by  calculating  the  interrater  reliability  across  tasks:  r*  = 
IMS.-MSrl/MS,  ] (Crocker  & Algina,  1986),  where  k is  the  number  of  judges  actually  providing  ratings. 
bSingle  rater  estimates  of  reliability  were  obtained  by  calculating  the  interrater  reliability  for  a single 
judge  across  tasks:  ii  = fMS.-MSrl/fMS.+fk-PMS./H. 

“Estimates  of  reliability  for  30  raters  were  obtained  by  calculating  the  interrater  reliability  for  thirty 
judges  across  tasks:  £30  = fMS,-MSrl/rMS,+fk-30,)MS./301. 


Table  16-2 

Reliability  of  Task  Ratings  Considering  Various  Recoding  Schemes:  Occupation-Specific  Tasks 


Type  of  Scale  and  Recoding  Scheme  Applied 
Frequency Importance 


Occupation 

la 

lb 

rr. 

la 

I b 

15005  Education  Administrators 

.97 

.95 

.95 

.95 

.85 

19005  General  Managers  & Top  Executives 

.99 

.99 

.98 

.98 

.96 

25 1 05  Computer  Programmers 

.87 

.85 

.77 

.86 

.82 

3 1 305  Teachers,  Elementary  School 

.99 

.99 

.97 

.98 

.98 

32502  Registered  Nurses 

.99 

.98 

.98 

.98 

.95 

32902  Medical  & Clinical  Laboratory  Technologists 

.90 

.88 

.86 

.89 

.84 

49008  Salespersons,  Except  Scientific  & Retail 

.97 

.91 

.96 

.97 

.90 

4901 1 Salespersons,  Retail 

.97 

.94 

.97 

.97 

.86 

49021  Stock  Clerks,  Sales  Floor 

.76 

.74 

.77 

.81 

.70 

49023  Cashiers 

.97 

.93 

.97 

.97 

.81 

51002  First  Line  Supervisors,  Clerical/Administrative 

.99 

.98 

.99 

.98 

.92 

53102  Tellers 

.95 

.93 

.88 

.94 

.86 

53905  Teachers'  Aides  & Assistants,  Clerical 

.87 

.92 

.66 

.83 

.87 

55108  Secretaries,  Except  Legal  & Medical 

1.00 

.99 

.99 

.99 

.96 

55305  Receptionists  & Information  Clerks 

.98 

.88 

.97 

.97 

.84 

55338  Bookkeeping,  Accounting,  & Auditing  Clerks 

.98 

.94 

.98 

.98 

.95 

55347  General  Office  Clerks 

1.00 

.99 

.99 

.99 

.96 

61005  Police  & Detective  Supervisors 

.96 

.95 

.91 

.93 

.91 

63014  Police  Patrol  Officers 

.99 

.99 

.98 

.98 

.97 

65008  Waiters  & Waitresses 

“.98 

.95 

.96 

.97 

.89 

65026  Cooks,  Restaurant 

.89 

.82 

.85 

.87 

.64 

65038  Food  Preparation  Workers 

.97 

.91 

.96 

.96 

.81 

66008  Nursing  Aides,  Orderlies,  & Attendants 

.97 

.90 

.97 

.97 

.88 

67005  Janitors  & Cleaners 

.99 

.98 

.98 

.98 

.93 

85132  Maintenance  Repairers,  General  Utility 

.96 

.93 

.95 

.96 

.91 

87902  Earth  Drillers,  Except  Oil  & Gas 

.95 

.91 

.95 

.94 

.78 

92974  Packaging  & Filling  Machine  Operators 

.77 

.21 

.76 

.78 

.42 

97102  Truck  Drivers,  Heavy  or  Tractor  Trailer 

.97 

.80 

.97 

.96 

.81 

97111  Bus  Drivers,  Schools 

.98 

.96 

.97 

.98 

.94 

Note.  Reliability  estimates  are  based  on  incumbents’  ratings  of  occupation-specific  tasks. 
Reliability  estimates  stipulated  as  £a  were  calculated  using  the  full  eight  point  scale  for  frequency, 
and  retaining  all  of  the  data  for  the  importance  scale.  Reliability  estimates  stipulated  as  jj,  were 
calculated  using  a reduced  seven  point  scale  for  frequency,  and  excluding  the  data  for  the 
importance  scale  where  the  rater  marked  not  relevant.  Reliability  estimates  stipulated  as  rc  were 
calculated  using  a binary  coded  scale  for  frequency  (relevant/not  relevant).  Decimals  are  omitted. 
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Table  16-2 

Reliability  of  Task  Ratings  Considering  Various  Recoding  Schemes:  Occupation-Specific  Tasks 


Type  of  Scale  and  Recoding  Scheme  Applied 
Frequency Importance 


Occupation £a  & £c  £a  lb 


15005  Education  Administrators 

.97 

.95 

.95 

.95 

.85 

19005  General  Managers  & Top  Executives 

.99 

.99 

.98 

.98 

.96 

25105  Computer  Programmers 

.87 

.85 

.77 

.86 

.82 

3 1 305  Teachers,  Elementary  School 

.99 

.99 

.97 

.98 

.98 

32502  Registered  Nurses 

.99 

.98 

.98 

.98 

.95 

32902  Medical  & Clinical  Laboratory  Technologists 

.90 

.88 

.86 

.89 

.84 

49008  Salespersons,  Except  Scientific  & Retail 

.97 

.91 

.96 

.97 

.90 

49011  Salespersons,  Retail 

.97 

.94 

.97 

.97 

.86 

49021  Stock  Clerks,  Sales  Floor 

.76 

.74 

.77 

.81 

.70 

49023  Cashiers 

.97 

.93 

.97 

.97 

.81 

51002  First  Line  Supervisors,  Clerical/Administrative 

.99 

.98 

.99 

.98 

.92 

53102  Tellers 

.95 

.93 

.88 

.94 

.86 

53905  Teachers'  Aides  & Assistants,  Clerical 

.87 

.92 

.66 

.83 

.87 

55108  Secretaries,  Except  Legal  & Medical 

1.00 

.99 

,99 

.99 

.96 

55305  Receptionists  & Information  Clerks 

.98 

.88 

.97 

.97 

.84 

55338  Bookkeeping,  Accounting,  & Auditing  Clerks 

.98 

.94 

.98 

.98 

.95 

55347  General  Office  Clerks 

1.00 

.99 

.99 

.99 

.96 

61005  Police  & Detective  Supervisors 

.96 

.95 

.91 

.93 

.91 

63014  Police  Patrol  Officers 

.99 

.99 

.98 

.98 

.97 

65008  Waiters  & Waitresses 

.98 

.95 

.96 

.97 

.89 

65026  Cooks,  Restaurant 

.89 

.82 

.85 

.87 

.64 

65038  Food  Preparation  Workers 

.97 

.91 

.96 

.96 

.81 

66008  Nursing  Aides,  Orderlies,  & Attendants 

.97 

.90 

.97 

.97 

.88 

67005  Janitors  & Cleaners 

.99 

.98 

.98 

.98 

.93 

85132  Maintenance  Repairers,  General  Utility 

.96 

.93 

.95 

.96 

.91 

87902  Earth  Drillers,  Except  Oil  & Gas 

.95 

.91 

.95 

.94 

.78 

92974  Packaging  & Filling  Machine  Operators 

.77 

.21 

.76 

.78 

.42 

97102  Truck  Drivers,  Heavy  or  Tractor  Trailer 

.97 

.80 

.97 

.96 

.81 

97111  Bus  Drivers,  Schools 

.98 

.96 

.97 

.98 

.94 

Note.  Reliability  estimates  are  based  on  incumbents’  ratings  of  occupation-specific  tasks. 
Reliability  estimates  stipulated  as  £a  were  calculated  using  the  full  eight  point  scale  for  frequency, 
and  retaining  all  of  the  data  for  the  importance  scale.  Reliability  estimates  stipulated  as  ij,  were 
calculated  using  a reduced  seven  point  scale  for  frequency,  and  excluding  the  data  for  the 
importance  scale  where  the  rater  marked  not  relevant.  Reliability  estimates  stipulated  as  rc  were 
calculated  using  a binary  coded  scale  for  frequency  (relevant/not  relevant).  Decimals  are  omitted. 
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Table  16-3 

Means  and  Standard  Deviations  of  Correlations  Between  Frequency  and  Importance  Scales 
Across  Tasks:  Occupation-Specific  Tasks 


Occupation na  Mr  SD, 


1 5005  Education  Administrators 

30 

•71 

.19 

19005  General  Managers  & Top  Executives 

20 

.77 

.09 

25105  Computer  Programmers 

18 

.78 

.08 

31305  Teachers,  Elementary  School 

17 

.69 

.19 

32502  Registered  Nurses 

21 

.80 

.10 

32902  Medical  & Clinical  Laboratory  Technologists1* 

15 

.86 

.29 

49008  Salespersons,  Except  Scientific  & Retail 

17 

.71 

.19 

4901 1 Salespersons,  Retail 

21 

.86 

.17 

49021  Stock  Clerks,  Sales  Floor 

10 

.87 

.05 

49023  Cashiers 

19 

.92 

.06 

51002  First  Line  Supervisors,  Clerical/Administrative 

19 

.75 

.12 

53102  Tellers 

13 

.61 

.30 

53905  Teachers'  Aides  & Assistants,  Clerical 

12 

.86 

.09 

55108  Secretaries,  Except  Legal  & Medical 

20 

.76 

.14 

55305  Receptionists  & Information  Clerks 

22 

.85 

.14 

55338  Bookkeeping,  Accounting,  & Auditing  Clerks 

21 

.85 

.07 

55347  General  Office  Clerks 

14 

.82 

.13 

61005  Police  & Detective  Supervisors 

20 

.63 

.12 

63014  Police  Patrol  Officers 

20 

.55 

.19 

65008  Waiters  & Waitresses 

20 

.70 

.26 

65026  Cooks,  Restaurant 

21 

.86 

.13 

65038  Food  Preparation  Workers 

21 

.83 

.07 

66008  Nursing  Aides,  Orderlies,  & Attendants 

18 

.85 

.10 

67005  Janitors  & Cleaners 

19 

.76 

.12 

85132  Maintenance  Repairers,  General  Utility 

18 

.78 

.10 

87902  Earth  Drillers,  Except  Oil  & Gas 

17 

.78 

.16 

92974  Packaging  & Filling  Machine  Operators 

19 

.91 

.06 

97102  Truck  Drivers,  Heavy  or  Tractor  Trailer 

13 

.75 

.24 

97111  Bus  Drivers,  Schools 

13 

.58 

.25 

Note.  All  correlations  were  calculated  based  on  the  ratings  assigned  by  raters  for  a given  job, 
task,  and  scale.  Frequency-importance  means  were  calculated  by  averaging  the  correlations  of  the 
frequency  and  importance  ratings  for  individual  tasks. 

“Number  of  correlations  averaged  (number  of  tasks),  not  number  of  observations  on  which 
correlations  were  calculated. 

bFor  this  occupation,  two  tasks  received  not  relevant  ratings  from  all  incumbents,  so  did  not 
contribute  to  the  mean  correlation. 
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Table  16-4 

Descriptor  Means  and  Standard  Deviations  on  the  Frequency  and  Importance  Scales  on  Two  Occupations:  Occupation-Specific 
Tasks 


ERIC 


<L> 

o 

C 

a 

ts 

o 

P 


> 

o 

c 

P 

3 

o 

.8 


SI 


SI 


CO 

Vi 

Cd 


co 

c 

o 

'■*—> 

Cd 

CU 

3 

O 

O 

O 


ON 

in 


oo 

CN 


CN 

CN 


O 

Os 


co 

P 


o 

cd 

TO 

P 

cd 

3 

00 

P 


<n  «> 

cn  .5 

IT  ° £ 
3 c 1 

CO  2 r 

w £ :§ 
^■g  * 

H cd  p 

D 60  g 

U S 3 

[Tl  In  3 

5 a 

^ co  G 


TO 

C 

cd 

co 

C 

0 
c3 

S3 

1 
E? 

O 

0 
00 

1 

TO 

c 

cd 

vi 

a 

o 


m .2 

c 


CL,  ^ _ 

O E p 

H « 1 

Q ^ c 

2 -S  « 
< ^ ■§ 
*S  « w 

Dh  W)  W 

w fa  £ 
O ■ "* 
< 

Z 

< 


00 

P 


o 


o 

Vi 

a 

o 


3 ^ 

S3  TO 

‘3  p 
§ * 

E?  p - 

o * 2 

P 


<u 

2 

oo 

P 


O 

a, 

Vi 

C 

« wM 

cd 

*3« 

X 

P 

TO 

C 

cd 


S 

OS 

W o 
2 2 & 
w ts  s & 

o £ c § 
oQ  25 


O CN  CN  O ON  CN  O O CN 

cn  © oo  Tfin  cn  cNcn 


cn  cn  oo 

H (N)  H 


O 

CN 


VO 


r-  oo 
^ CN 


VO  CN 
cn  vo 


VO  CN 
CN 


Tf 


r-*  m 


cn  o vo 
cn  m o 


Tf 

CN 

ON 

ON 

r- 

00 

© 

cn 

in 

vq 

CN 

cn 

CN 

cn 

Vi 

C 

.2 
**— * 
cd 
IN 

*3 

cd 

oo 


oo  m cn 
cn  vo  ^ 

CN  CN  ^ 


P 

E 

§ 

C/5 

C 

o 

cd 

.2 

’3 

cd 

oo 


o -3 


p 

SI 


co 

c 

o 

cd 

Ui 

P 

Cu 

O 

00 

G 

*3 

c 


oo 

P 

> 

O 

P 

S’ 

o 


cd  O 


C 

o 

o 

P 


co 

C 

p 


Vi 

p 


> c 
*o  o 

O -*3 

« td 
oo  H 
.£  c 
•*—>  cd 
cd  oo 

c Uh 

’-3  ° 

u P 
O X5 
O ~ 

TO  2 
C 
cd 


p 


TO 
G 

cd 

W3 

.2 

O 

c 

p 
00 
cd 

B 

Cd  C/5 


Vi 

Q* 

O 

-G 

co 

U 

O 

£ 

TO 

G 

cd 

oo 

c9 

G 


Vi 

04 

15 

C/5 

c 

3 

13 

U 

00 

c 

a 

U 

O 

& 

P 

> 


co 

<D 

> 

*4—* 

O 

<D 

S’ 

o 

E 

a 

Ui 

00 

2 "3 

g 


fe  p - 

G rrt 


a > 


cd  ^ 


(U 


w 

C 

V 


o 

% 

o 


Vi 

G 

O 

vi 


E 

00 

>> 

Vi 


E 

cd 

Ut 

00 

o 


oo 


W o 


oo 

2 

3 

T3 

<D 

O 

o 

i-l 

cx 

HD 


3 O = 
T3  <D  .5 
g T3  Q. 

O*  g>  S 

U .S 

8 -i  •g 
I a 8 

G G <L> 


? 


o 

cd 

c^ 

o 


G -2 
cd 


Sll'ssgfl's 

n O r?  oo  t-1  cd 


CN 


C Cd 
cd  C 

*3 

P u 

:s  8 

0 ° 

CO 

O4  TD 

Id 

3 

co  P 

G4 

TD 

-2  8 

’> 

13  0 
> ^ 

-5 

p .a 

c 

Q Q 

cn 

* 

00 

<4— » 

G 

<D 


o a 
*3 
5 


2 

<D  00 

•o  c 
tp  -a 
^ c 
•C  o 

X 5 
? ^0) 
c/5  Ou 

c £ 

V — * 
£ "2 
8 g 

M Ml 
« .£ 
T3  ’G 
C C 

w <2 
£ U 

g -° 


<u 

CO 

TD 

C 

cd 

cu 

X 

<D 

T3 

C 

cd 

CO 

CO 

<L> 

C 

Vi 

3 

X) 

o 

-4-* 

o 

E 

2 

a, 

o 

CO 


<D 

CO 

CO 

O 

O 

G 

<l> 

Un 

,<L> 

c*-< 

G 

O 

o 

oo 

c 

u 

3 

TD 

C 

o 

o 

.5 

0) 

o 

c 

cd 


2 

o 

04 

o 

o 

o 

.5 

*3 

c 


T3 

c 

cd 

CO 

T3 

C 


o 

C/5 


TD 

C 

cd 

co" 

<L) 

0 

1 

<D 

CO 

8 

o 

5 

o 

Ui 

GU 

O 

<4— I 

co 

G 

O 


O c 
Oo  a> 
> 
G 


a b o 

w to  O 


S 0/5  •§ 

> G S 

G ~ 

c * 8 

§ a -g 

co  *3  *2 

Hi 

£uw 


n *> 
O “ 

° 3 

8 x> 

03  8 

*G  ^ 

o S 

60 

« 2 
2 cu 


cd 

o 

c 

JG 

O 

8 

CO 

<D 

T3 


cd 

5 c/3 

w 

o S 

cd 

<L>  On 

.2  o 

SI 

CO  O4 

>%  c 

•E-2 

3 Cd 

5 .a 

5 c 

P cd 

O GO 
O t-T 
co  O 

a-g 

C cd 
cd  m 

00  .S 


co 

C O 
o C 

■g  .2 

fe  g 

Q-  5 

O cd 

2 » 

00  T3 

.5  S 

"rrt 

cd  ~ 

S.E 

co  G 

’C  ’3 
o i- 
04  ^ 

2 BO 

C/5  .2 


co 

00 

c 

■4—* 

<D 

<D 

E 


CO 

8 

13 

ts 

cd 

co 


o ^ 


p *3 
<D  SZ 
vi 

8 «> 
04 .5 


in 

CN 


o 

r- 


00  vo 

cn  o 


r-  cn  o 

00  m vo 

© CN  © 


co 

<D 

O 

•rj 

t o 

8 ’p 
00  cd 
G O 
*G  X) 

P * 


G 

<D 

5 

<D 

> 

O 

u< 

.1 

T3 

C 

<D 

5 

5 

o 

o 

<D 


.2  o 

I '2 

O 0> 

O4  CO 

O £ 
4— » co 

cd  O 

"3  .5 

SCO 

3 

5 ^ 


t: 

o 

04 

04 

3 

co 

T3 

C 

cd 

<D 

4—4 

O 

5 

2 

Cu 

o 

co 

<D 

00 

c 

cd 

x: 

o 

CO 

C 

<u 

5 

5 

o 

o 

o 


<u 

te  .2 

cd  > 
co  P 

i « 
a o 

S3  ^ 

a 

o t; 

■g  § 

§ o 

. c 

CO 

8 "5 

a £ 


CO 

04 


i 2 

CU  P. 


T3  co 
C P 
cd  ~ 

co  2 

2 o 

Cd  --g 

p^  s 

P o 


»■§ 

TD  q. 

C 

cd  00 

S I 

P G q u-r 

cl-  « ft-  S Q 


T3 
C 

cd 

co 

P 

*3  CO 
tt  Cd 

o B 

2S-S 

w <D 

2 w> 

P g 

> 2 

• ^ *G 

P P 


co  e 

00  c 

C cd 


•O 

c 


£J  00 
•O  Id 

§ *5 

” p 

>>  04 

O CO 

s ■g 

o<  « 

.2  c 

S -2 
B.JS 


i 

v 

CO 

P 


00 

2 

&. 

vi 

a 

o 

4— • 

g 


<D  P 


X 

P 


c 

G 

O 

co 

t-i 

P 

04 


s a 

G 

O cd 
*cd 

-2  c 

CO  P 

•5b  « 
J2 

8 S3 
>?  2 
g & 

<-S 


c/T  T3 

V C 

'ft!  « 

> e« 
•Z3  M 
o c 

£3  -C 
60  « 
•s  a 

P Ui 

P O 
P CO 

C C ■ 

o o 

o 

Sv 

t:  ,2P 
O -S 
pu  g 


04  p 


K £ 

co  .G 


5 

x> 

3 

CO 

T3 


in  \D  S 00  ON  o ^ 


cn  cn  n- 


in 


vo 


r-  00 


: . & 


m 

00 

vo  m 

© 

ON 

© 

CN 

© 

in 

CN 

Tl" 

cn 

00 

ON 

r- 

in  00 

NO 

cn 

in  © 

© 

© 

cn 

in 

© 

CN 

in 

in 

00 

© l> 

CN 

cn 

cn 

cn 

CN 

cn 

cn 

CN 

CN 

cn 

cn 

CN 

CN 

cn  < 

02 

in 


Table  16-4  (continued) 

Descriptor  Means  and  Standard  Deviations  on  the  Frequency  and  Importance  Scales  on  Two  Occupations:  Occupation-Specific 
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Table  16-4  (continued) 

Descriptor  Means  and  Standard  Deviations  on  the  Frequency  and  Importance  Scales  on  Two  Occupations:  Occupation-Specific 
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Chapter  17 

Conclusions  and  Recommendations 


Norman  G.  Peterson 
American  Institutes  for  Research 


As  we  stated  in  Chapter  2,  the  initial  0*NET  data  collection  effort  was  not  intended  to 
provide  a comprehensive  description  of  all  jobs  in  the  United  States  economy,  but  it  was 
intended  to  be  a prototype  for  a system  that  could  lead  to  such  a descriptive  system.  In  this  report 
we  have  focused  on  the  occupational  sampling  techniques,  data  collection  procedures,  and 
analyses  of  the  taxonomic  measures  of  the  0*NET.  A companion  report,  Q*NET:  An 
Information  System  for  the  Workplace.  Designing  an  Electronic  Infrastructure  (Rose,  Hesse, 
Silver,  & Dumas,  1996)  describes  the  development  of  the  electronic  database  intended  to  house 
and  make  available  0*NET  information,  as  well  as  associated  tools  for  taking  advantage  of  that 


information. 


17-2  Peterson 

Chapter  15  discusses  many  of  the  implications  of  the  data  analysis  results  for  possible 
0*NET  applications.  Here  we  provide  further  summaries  and  some  addtional  comments  on  the 
results,  and  discuss  possible  future  directions  for  the  0*NET  system. 

Review  of  Analyses 
Reliability 

The  primary  statistic  computed  here  was  the  interrater  agreement  coefficients.  Figure  17- 
1 summarizes  the  results  for  the  nine  domain  questionnaires.  It  shows  the  observed  coefficients 
(as  extracted  from  table  5 for  most  of  the  questionnaires)  which  are  based  on  approximately  10 
raters  per  occupation  for  most  domains,  and  estimates  for  the  case  for  30  raters,  which  was  our 
original  view  of  the  desired  number  of  raters  within  each  occupation.  The  observed  coefficients 
are  all  at  least  .70,  except  for  organizational  context,  occupational  values,  the  dichotomous  job- 
entry  rating  in  skills  and  the  importance  scale  for  work  styles.  Possible,  somewhat  differing 
reasons  for  these  lower  than  desired  results  are  discussed  in  the  respective  chapters,  but  the  major 
conclusion  to  be  drawn  from  these  results  is  that  the  domain  questionnaires  produce  reliable 
results  when  completed  by  incumbents.  The  estimated  coefficients  for  30  raters  are  all  .79  or 
greater,  with  most  in  the  .90’ s,  illustrating  that  30  is  a sufficient  number.  For  many  of  the 
domains,  fewer  raters  should  be  sufficient,  based  on  these  results.  While  30  should  be  the  goal, 
fifteen  seems  to  be  a reasonable  number  for  a minimum  number  of  raters  for  each  questionnaire 
per  occupation.  Those  questionnaires  that  showed  lower  results  should  be  held  to  higher 
minimums  and  possible  reasons  for  the  lower  results  should  be  pursued  and,  if  possible, 
corrected. 

Figure  17-2  shows  similar  results  for  aggregated  descriptor  scores.  Aggregate  scores  were 
formed  by  computing  mean  values  for  the  descriptors  that  were  categorized  into  the  second-level 
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of  the  domains’  hierarchies.  As  shown,  this  was  not  appropriate  for  some  domains.  These  results 
mirror  those  in  Figure  17-1,  except  that  the  coefficients  are  generally  a bit  higher,  with  a few 
exceptions.  These  results  certainly  support  the  use  of  aggregate  or  higher-level  scores  for 
describing  occupations. 

Both  figures  show  that  the  level  and  importance  scales  used  in  five  domains  are 
approximately  equal  in  terms  of  reliability.  However,  this  finding  must  be  tempered  by  the  fact 
that  the  two  scales  were  administered  in  sequence,  with  importance  following  level.  The  use  of 
both  scales  in  this  way  appears  to  lead  to  reliable  results  for  both;  dropping  one  or  the  other 
might  reduce  the  level  of  reliability  for  the  retained  scale.  McCormick  (1964)  found,  years  ago, 
that  use  of  multiple  scales  tends  to  increase  the  reliability  of  all  the  scales. 

Scale  Relationships 

Figure  17-3  shows  the  correlations  between  scale  types,  computed  in  two  ways,  for  those 
domains  where  multiple  scale  ratings  were  obtained.  Level  and  importance  scales  were  used  in 
five  domains  and  the  correlations  ranged  from  .82  to  .96  for  the  across  occupations  correlations 
and  from  .90  to  .95  for  the  across  descriptors  correlations.  These  are  very  high  correlations  and 
raise  the  possibility  that  redundant  information  is  being  obtained.  However,  the  standard 
deviations  of  the  mean  correlations  show  that  there  is  a considerable  amount  of  variance  across 
occupations  in  the  correlations,  when  compared  to  the  correlations  across  descriptors.  The 
standard  deviations  for  the  across  occupations  correlations  are  .04,  .2 1 , . 1 1 , .30  and  .11  for  skills, 
knowledges,  generalized  work  activities,  abilities,  and  work  styles,  respectively.  Comparable 
across  descriptors  correlations  are  .04,  .04,  .05,  .07,  and  .05.  This  means  that  the  relationship 
between  the  level  and  importance  scales  varies  considerably  across  occupations  for  four  of  the 
five  domains,  which  argues  for  the  retention  of  both  scales.  On  a practical  note,  completion  of  the 
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importance  scale  takes  very  little  time  once  the  descriptor  definition  has  been  read  and  the  level 
rating  has  been  completed.  We  would  argue  strongly  that  the  level  scale  should  be  retained 
because  of  the  additional  information  provided  by  the  level  scale  anchors.  Given  these  findings, 
and  our  belief  that  the  level  scale  certainly  should  be  retained,  there  may  be  relatively  little  time 
savings  (on  the  respondent’s  part)  should  the  importance  scale  be  dropped  from  further  use. 

The  other  scales,  job  entry  requirement  in  skills  and  frequency  in  generalized  work 
activities,  show  somewhat  lower  correlations  with  level  and  importance  than  those  two  scales  do 
with  themselves.  The  job  entry  correlations  range  from  -.66  to  -.74  (lower  score  = not  required  at 
entry),  evidently  providing  some  unique  information.  The  frequency  scale  scores  also  appear  to 
provide  somewhat  less  redundant  information,  but  its  correlations  with  importance  and  level 
scale  scores  still  range  from  .82  to  .91. 

Descriptor  Relationships 

To  assess  the  internal  structure  of  the  descriptor  variables,  correlations  were  computed  for 
and  principal  components  analysis  conducted  on,  primarily,  the  mean  occupation  scores  on  the 
level  scales.  These  analyses  are  perhaps  more  affected  by  the  number  of  occupations  available  to 
enter  the  analysis  than  are  many  of  the  other  analyses.  These  analyses  are  too  numerous  and 
complex  to  be  efficiently  summarized  here,  but  they  generally  provided  intuitively  sensible 
results  within  each  domain.  As  discussed  in  Chapter  15,  these  analyses  were  not  intended  to 
exactly  duplicate  the  a priori  hierarchical  structure  of  the  domains,  but  were  expected  to  be 
consistent  with  them,  and  were,  for  the  most  part. 

The  a priori  hierarchical  structure  was  examined  more  directly  in  Chapter  15,  particularly. 
Table  15-1.  This  table  showed  that  the  mean  within-category  correlations  were  higher  than  the 
mean  across-category  correlations  for  descriptor  scores  in  each  of  the  six  domains  for  which  such 
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statistics  could  be  computed,  although  just  barely  so  for  skills  and  occupational  values.  These 
findings  support  the  use  of  the  category  schemes  for  forming  aggregate  scores  for  the  0*NET, 
but  with  a word  of  caution  for  the  skills  and  values  domains. 

Occupation  Differences  and  Discriminant  Analyses 

The  two  kinds  of  analyses  conducted  here  supported  the  contention  that  the  0*NET 
system  is  useful  for  differentiating  occupations.  Comparisons  of  six  occupations  across 
descriptor  profiles  showed  intuitively  sensible  results,  and  the  discriminant  function  analyses 
identified  the  structure  of  descriptors  within  each  domain  that  appeared  to  be  accounting  for  the 
differentiation  between  occupations.  In  addition,  the  multivariate  analyses  of  variance  using 
occupations  and  descriptors  as  facets  showed  that  the  occupation  by  descriptor  interaction  was 
generally  statistically  significant,  again  showing  that  the  0*NET  data  are  useful  for 
differentiating  occupations. 

Convergence  With  Analysts’  Ratings 

We  completed  these  analyses  for  five  of  the  nine  content  domains,  and  some  of  those 
results  are  summarized  in  Figure  17-4.  Generally,  the  pattern  of  ratings  was  similar  across  the 
two  rater  types;  the  correlations  of  the  mean  ratings  provided  by  the  two  raters  ranges  from  .53  to 
.74.  The  d2  values  range  from  .47  for  the  five -point  importance  scale  in  knowledges  to  1.51  for 
the  eight-point  level  scale  in  skills.  When  the  square  roots  of  these  values  are  taken,  the  average 
differences  are  all  reasonably  close  to  one  scale  point  for  the  eight  point  level  scales  and  between 
a half  and  one  scale  point  for  the  five-point  importance  scale.  These  do  not  seem  like  practically 
large  differences,  but  it  must  be  kept  in  mind  that  these  are  average  values.  For  some  occupations 
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there  could  be  more  substantial  differences.  We  conclude,  however,  based  on  these  findings  that 
there  is  substantial  agreement  between  the  two  types  of  ratings,  sufficient  to  warrant  the  interim 
use  of  analyst  ratings  in  anticipation  of  the  future  availability  of  incumbent  ratings. 

These  findings  also  raise  the  possibility  of  combining  the  two  kinds  of  ratings  for 
purposes  of  describing  occupations.  These  data  do  not  seem  to  preclude  such  a procedure,  nor  do 
they  argue  for  complete  substitutability.  Furthermore,  some  of  the  domains  were  not  rated  by 
analysts  because  it  was  judged  that  the  stimulus  material  provided  to  analysts  would  be 
insufficient  for  rating  those  domains.  These  points  support  the  notion  that  data  from  the  two 
sources  must  be  very  carefully  combined  to  describe  occupations. 

Additional  Validity  Evidence  and  Other  Analyses 

Most  of  the  domain  chapters  (3  through  1 1)  present  additional  analyses  intended  to 
evaluate  the  construct  validity  of  the  measures.  In  addition,  chapters  12-16  present  analyses  that 
further  evaluate  the  meaningfulness  and  usefulness  of  the  0*NET  system.  The  authors  of  each  of 
these  chapter  have  drawn  their  own  conclusions  and  they  will  not  be  repeated  here.  However, 
some  comments  are  in  order. 

Chapter  12  discussed  the  possible  sources  of  variance,  both  desired  and  undesired,  in 
occupational  analysis  ratings  and  summarized  the  evidence  available  on  this  topic.  Many  issues 
are  yet  to  be  addressed  in  this  area;  some  are  probably  not  practically  feasible  to  address  in  the 
forseeable  future.  Nevertheless,  this  chapter  provides  a coherent  organization  of  the  issues  and 
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Chapter  13  discusses  the  relationships  between  measures  across  the  0*NET  domains. 
Several  approaches  for  analyzing  these  relationships  were  employed;  together  they  provide  strong 
evidence  for  the  construct  validity  of  the  0*NET  descriptor  system.  Expected  relationships 
occurred  where  they  should,  and  did  not  occur  where  they  were  not  expected.  Future  analyses  in 
this  area  could  focus,  among  other  things,  on  making  estimates  of  scores  in  one  domain  from 
scores  in  another.  Such  analyses  are  of  obvious  practical  use,  but  are  at  present  limited  by  the 
number  of  occupations  for  which  incumbent  analyses  are  available.  This  is  not  a hindrance  for 
the  occupational  analyst  database,  however,  since  data  are  available  for  all  1,122  0*NET 
occupational  units  on  five  of  the  major  content  domains.  A replication  of  the  analyses  done  using 
the  incumbent  data  on  the  analyst  data  would  provide  a gauge  for  the  advisability  of  using  the 
analyst  data  to  provide  such  estimates. 

The  analyses  presented  in  Chapter  14  show  that  the  skill  and  generalized  work  activities 
domains  are  effective  in  forming  occupational  clusters.  The  analyses  explored  the  impact  on 
cluster  solutions  of  type  of  variable,  clustering  procedure,  and  sample  (incumbent  data  or  analyst 
data),  as  evidenced  by  various  evaluative  statistics  and  the  similarity  of  cluster  solutions.  Briefly 
put,  different  types  of  variable  did  produce  somewhat  different  solutions,  as  one  would  hope,  but, 
changes  in  clustering  procedure  did  not  seem  to  have  an  appreciable  effect,  within  the  confines  of 
these  investigations.  There  were  some  differences  between  the  solutions  based  on  analysts  vs. 
incumbents,  but  these  were  not  large  and  became  smaller  when  more  descriptors  were  used  to 
make  the  clusters. 

A number  of  implications  of  the  findings  of  these  analyses  are  discussed  with  regard  to 
particular,  possible  uses  of  the  0*NET  system  in  Chapter  15,  but  a primary  focus  is  on  the 
development  of  methods  for  forming  job  descriptions.  There  are  some  thorny  issues  here,  but 
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they  appear  to  be  eminently  solvable  and  are  partially  dependent  on  the  particular  purposes  for 
forming  job  descriptions. 

Chapter  16  presents  analyses  of  occupation-specific  data,  particularly  frequency  and 
importance  ratings  of  tasks.  These  analyses  show  that  such  ratings  had  high  interrater  agreement 
coefficients  in  general,  with  30-rater  estimates  of  .90  or  greater  for  all  but  four  of  the  29 
occupations.  Much  of  that  chapter  describes  procedures  that  have  been  developed  and  tried  out 
for  gathering  occupation-specific  data  within  the  structure  of  the  0*NET  content  domain, 
particularly  the  generalized  work  activities  and  skills. 

Comments  on  Future  Directions 

The  authors  of  the  various  chapters  have  discussed  possible  avenues  for  future  research 
and  investigation.  Some  broader  comments  are  made  here. 

With  regard  to  methods  of  collecting  data,  the  paramount  lesson  that  has  been  learned  is 
that  multiple  methods  should  be  employed.  That  data  collection  should  be  an  ongoing  0*NET 
activity  is  obvious;  less  obvious  is  how  that  activity  should  be  organized  and  carried  out.  This  is 
at  least  as  much  an  organizational  issue  as  it  is  a technical  issue,  but  some  guidance  can  be 
offered  based  on  our  experiences  over  the  first  two  years  of  0*NET  development.  First,  if  a 
variety  of  approaches  are  to  be  used  to  collect  data,  then  a common  set  of  demographic  or 
identifying  information  must  be  collected.  While  the  exact  makeup  of  this  set  of  variables  is  not 
fully  known,  it  includes  the  obvious  personal  identifiers  (race,  sex,  ethnicity,  age),  but  should 
also  include  other  information  about  the  organization  or  establishment  in  which  the  respondent  is 
situated.  These  variables  are  essential  to  allow  the  data  to  be  combined  across  the  methods  to 
yield  an  accurate  description  of  occupations.  Second,  some  sort  of  incentives  must  be  found  to 
assist  in  the  collection  of  data.  The  most  mundane  would  be  direct  payment  of  respondents,  but 
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this  is  often  not  feasible  nor  even  the  most  effective  method  of  securing  cooperation.  To  collect 
data,  cooperation  must  often  be  secured  at  many  levels— the  organization  housing  the 
occupational  incumbents,  including  home  and  branch  office  management,  supervisors  of 
incumbent  respondents,  employer  and/or  employee  associations,  unions,  supervisors,  etc.  Each  of 
these  entities  calls  for  slightly  different  approaches  to  securing  cooperation  and,  thus,  different 
kinds  of  incentives.  As  was  demonstrated  in  the  prototype  data  collection,  success  must  be 
secured  at  all  these  levels  to  insure  successful  data  collection.  We  believe  that  the  most  powerful 
incentive  of  all  will  be  a highly  useful  0*NET  system  with  multiple,  tailored  applications 
making  use  of  the  0*NET  database.  These  applications  provide  tangible  evidence  to  users  of  the 
“fruits  of  labor”  of  their  cooperation  in  data  collection.  Although  the  prototype  project  has  moved 
0*NET  a long  way  toward  this  goal,  much  work  remains  to  be  done  in  this  arena. 

This  last  comment  leads  to  a recommendation  for  a possible  way  to  organize  the  0*NET 
effort,  both  research  and  development  and  ongoing  data  collection  and  product  development. 
Following  some  private  sector  models,  0*NET  representatives  could  become  familiar  with  data 
collection  methods  and  procedures  and  the  available  0*NET  applications.  Armed  with  this 
knowledge  of  the  benefits  of  0*NET  that  could  be  applied  in  various  settings,  they  would  be  in 
excellent  position  to  recruit  cooperation  from  organizations  and  employee  associations— and 
would  have  the  knowledge  to  assist  in  the  technical  details  of  data  collection.  Furthermore,  these 
representatives  might  specialize  in  particular  industries  or  broad  occupational  groupings, 
therefore  becoming  much  more  knowledgeable  in  the  core  activities  undertaken  in  the  industry  or 
occupational  area  of  their  specialty.  These  “sales  representatives”  could,  if  more  detailed 
knowledge  is  required,  call  on  a central  core  of  0*NET  technical  representatives  that  would  be 
working  day-to-day  with  the  details  of  data  collection  or  with  monitoring,  cataloguing,  and/or 
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developing  particular  0*NET  applications.  Collectively,  these  technical  representatives  would 
perform  the  ongoing  research,  data  quality,  and  technical  control  functions  necessary  to 
continually  update  and  improve  the  0*NET  system. 

Such  an  organizational  scheme  is  only  one  possible  way  to  proceed,  of  course,  but  it  does 
attempt  to  combine  the  working  knowledge  of  the  “input”  and  “output”  functions  of  the  0*NET 
system  in  a cadre  of  “sales  representatives”  so  that  the  benefits  of  participating  in  0*NET  data 
collection  are  tightly  coupled  with  the  demand  on  resources  to  engage  in  that  participation.  As 
0*NET  grows  in  use,  then  collections  or  associations  of  particular  types  of  users  could  be 
formed,  perhaps  even  resulting  in  an  annual  convention  for  0*NET  researchers,  data  providers; 
and  application  users. 

A comment  is  in  order  about  occupational  unit.  A major  effort  on  the  part  of  the 
Occupational  Analysis  Field  Centers  has  been  the  identification  of  approximately  1122 
occupational  units,  which  are  an  expansion  of  the  Occupational  Employment  Statistics  (OES)  set 
of  occupations.  These  units  were  identified  through  a thorough  and  careful  analysis  of  Dictionary 
of  Occupational  Titles  (DOT;  U.S.  Department  of  Labor,  1991)  information.  The  0*NET  system 
presently  employs  these  units  as  their  basic  occupational  set,  and  they  are  also  being  considered 
as  part  of  a larger  Federal  effort  to  revise  the  Standard  Occupational  Classification  system.  The 
future  may  bring  additional  changes  to  the  makeup  of  the  basic  set  of  occupations.  One  of  the 
advantages  of  the  0*NET  system,  with  its  multiple  domains  and  measures,  is  its  capability  of 
forming  alternative  groupings  of  occupations,  as  demonstrated  in  Chapter  14,  thus  allowing  the 
use  of  such  alternative  sets  for  special  uses.  However,  a common  set  of  occupational  units  with 
some  degree  of  stability  is  essential  for  combining  data  across  different  data  collection  efforts— 
such  as  those  conducted  by  the  Bureau  of  Labor  Statistics.  Therefore,  attention  to  the  technical 
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and  practical  utility  of  the  occupational  unit  scheme  used  by  0*NET,  and  its  coordination  with 
other  Federal,  state,  and  private  sector  data  collection  efforts  is  vitally  important. 

Each  content  domain  and  its  measures  have  been  separately  analyzed  and  reported  on  in 
earlier  chapters.  Although  some  domains  clearly  have  stronger  results  supporting  their  continued 
use,  collectively,  all  the  domains  seem  technically  sufficient  enough  for  continued  inclusion  in 
the  0*NET  system.  Some  fine  tuning  in  the  makeup  of  a few  of  the  domains  is  in  order  and 
should  be  undertaken,  but  these  changes  are  not  viewed  as  particularly  urgent.  Once  again,  the 
relatively  limited  number  of  occupations  available  for  analysis  tempers  our  conclusions  in  this 
regard.  With  more  data,  it  might  become  more  apparent  that  one  or  more  domains  should  be 
drastically  revised  or  dispensed  with  from  a technical  point  of  view.  However,  it  is  more  likely 
that  such  decisions  will  be  driven  by  the  uses  to  which  the  data  are  eventually  put,  or  not  put.  It 
must  be  kept  in  mind  that  each  of  these  domains  has  associated  with  it  a potential  set  of 
interested  consumers,  based  on  prior  user  surveys  and  contacts  with  potential  users  throughout 
this  project.  As  applications  are  developed  and  put  into  use,  the  usefulness  of  the  various 
measures  in  the  0*NET  system  will  become  more  clear.  If  a domain  is  clearly  useful,  but  has 
some  technical  problems,  the  first  order  of  business  would  be  to  fix  the  technical  problems.  On 
the  other  hand,  a technically  adequate  domain  that  finds  no  user  base,  even  after  some  amount  of 
time,  is  clearly  a candidate  for  deletion.  Likewise,  it  may  be  that  candidates  for  addition  to  the 
0*NET  content  model  will  become  noticed  in  the  same  way. 

A Final  Note 

We  think  a tremendous  degree  of  technical  and  practical  progress  has  been  made  on  the 
prototype  0*NET  project.  Indeed,  we  think  the  project  has  resulted  in  the  production  of  an 
occupational  database  that  is  of  some  immediate,  practical  use,  perhaps  somewhat  beyond  the 
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initial  expectations  and  hopes  of  those  of  us  most  closely  involved  in  the  project.  This  report, 
with  its  companion  report  (Rose,  Hesse,  Silver,  and  Dumas,  1996)  and  the  earlier  content  model 
report  (Peterson,  Mumford,  Borman,  Jeanneret,  and  Fleishman,  1995)  stand  as  the  primary 
documentation  for  the  completed  work.  Much  good  work  has  been  done,  but  much  remains  to  be 
done. 
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Figure  17-1 

Interrater  Agreement  Coefficients  for  Each  Scale  Type 


Questionnaire 

Scale 

Ek 

£30 

Skills 

Level 

.79 

.93 

Importance 

.79 

.93 

Job  Entry  Requirement 

.60 

.83 

Knowledges 

Level 

.86 

.95 

Importance 

.85 

.94 

Training,  Education,  Licensure,  & 
Experience 

Instructional  Program 

.78 

.92 

Educational  Subject  Area 

.74 

.90  . 

Licensure 

.85 

.95 

Experience 

.79 

.93 

Generalized  Work  Activities 

Level 

.80 

.92 

Importance 

.78 

.92 

Frequency 

.74 

.90 

Work  Context 

.87 

.95 

Organizational  Context 

Across  Occupations 

.64 

.84 

Across  Organizations 

.45 

.79 

Abilities 

Level 

.82 

.93 

Importance 

.82 

.93 

Occupational  Values 

.60 

.82 

Work  Styles 

Level 

.70 

.88 

Importance 

.67 

.86 

Note.  & is  the  observed  interrater  agreement  coefficient;  |jo  is  the  estimated  interrater  agreement 
coefficient  for  30  raters. 
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Figure  17-2 

Interrater  Agreement  Coefficients  for  Aggregate  Descriptors  for  Each  Scale  Type 


Questionnaire 

Scale 

r k 

E30 

Skills 

Level 

.86 

.95 

Importance 

.86 

.95 

Job  Entry  Requirement 

.69 

.88 

Knowledges 

Level 

.85 

.94 

Importance 

.85 

.94 

Training,  Education,  Licensure,  & 
Experience 

Instructional  Program 

n/a 

n/a 

Educational  Subject  Area 

n/a 

n/a 

Licensure 

n/a 

n/a 

Experience 

n/a 

n/a 

Generalized  Work  Activities 

Level 

.86 

.95 

Importance 

.84 

.94 

Frequency 

.78 

.92  • 

Work  Context 

n/a 

n/a 

Organizational  Context 

Across  Occupations 

.60 

.82 

Across  Organizations 

.41 

. .77 

Abilities 

Level 

.86 

.95 

Importance 

.82 

.93 

Occupational  Values 

.57 

.81 

Work  Styles 

Level 

.76 

.91 

Importance 

.73 

.89 

Note:  ik  is  the  observed  interrater  agreement  coefficient;  ijo  is  the  estimated  interrater  agreement 
coefficient  for  30  raters. 
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Figure  17-3 

Mean  Correlations  Between  Scales  Across  Occupations  and  Descriptors 


Questionnaire 

Scales 

Across 

Occupations 

Across 

Descriptors 

Skills 

Level/Importance 

.96 

.95 

Level/Job  Entry  Requirement 

-.71 

-.66 

Importance/Job  Entry 
Requirement 

-.74 

-.69 

Knowledges 

Level/Importance 

.90 

.95 

Training,  Education, 
Licensure,  & Experience 

n/a 

n/a 

Generalized  Work 
Activities 

Level/Importance 

.93 

.92 

Level/Frequency 

.88 

.82 

Importance/Frequency 

.91 

.89 

Work  Context 

n/a 

n/a 

Organizational  Context 

n/a 

n/a 

Abilities 

Level/Importance 

.82 

.92 

Occupational  Values 

n/a 

n/a 

Work  Styles 

Level/Importance 

.93 

.90 

Note:  Across  occupations  correlations  are  the  mean  ratings  on  a given  occupation  for  all 
descriptors  for  one  scale  correlated  with  the  mean  ratings  on  the  same  occupations  for  all 
descriptors  on  the  other  scale,  averaged  across  occupations.  Across  descriptors  correlations  are 
the  mean  ratings  for  a given  descriptor  for  all  occupations  for  one  scale  correlated  with  the  mean 
ratings  for  that  descriptor  for  all  occupations  for  the  other  scale,  averaged  across  descriptors. 


3 


O 


Figure  17-4 

Mean  Correlations  and  Squared  Differences  Between  Incumbents’  and  Analysts’  Ratings 


Questionnaire 

Scale 

5a  

d2 

Skills 

Level 

.74 

1.51 

Importance 

.67 

.72 

Knowledges 

Level 

.65 

.86 

Importance 

.65 

.47 

Training,  Education,  Licensure,  & 
Experience 

n/a 

n/a 

Generalized  Work  Activities 

Level 

.71 

1.09 

Importance 

.61 

.82 

Frequency 

.53 

n/a 

Work  Context 

.64 

n/a 

Organizational  Context 

n/a 

n/a 

Abilities 

Level 

.70 

.99 

Importance 

.65 

.39 

Occupational  Values 

n/a 

n/a 

Work  Styles 

n/a 

n/a 

Note:  £ia  is  the  mean  correlation  between  incumbent  and  analyst  mean  occupation  ratings,  d'  is 
the  mean  squared  difference  between  incumbent  and  analyst  mean  occupations  ratings. 
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ORGANIZATIONAL  INFORMATION  • 


ORGANIZATION  ID: ADDRESS: 

CONTACT: 

TITLE:  

ORGANIZATION  NAME: TELEPHONE  #: 

SECONDARY  NAME:  CONTACT  PHONE: 

SIC:  

OAFC  SCREENER  INFORMATION 

NEGOTIATOR: OAFC  SCREENER  DATE: 


[ASK  TO  SPEAK  WITH  CONTACT  ABOVE.] 


START.  We  would  like  to  start  by  asking  you  to  provide  some  organizational  information.  Then,  we  can 
discuss  how  you  will  help  us  collect  data  from  your  employees. 

To  begin,  we  would  like  to  know  a little  about  you. 


SI.  How  many  years  have  you  worked  for  this  organization? years  (IF  LESS  THAN  1,  ENTER 

0) 


S2.  How  many  years  have  you  held  your  current  position? years  (IF  LESS  THAN  1,  ENTER 

0) 


S3.  What  is  your  current  job  title? 


Next,  I will  be  asking  some  questions  about  your  organization.  Most  of  the  questions  will  be  about 
your  organization  at  the  following  location:  [STREET  NAME].  However,  some  questions  will  refer  to  your 
entire  organization. 


1 . Were  you  in  existence  before  1990? 

1.  Yes 

2.  No 


NOTE  TO  CAII  PROGRAMMER:  IF  RESPONDENT  ANSWERS  NO  TO 
Ql.,DO  NOT  ASK  Q13,  Q19,  Q24-Q27.  \ : :■  -Y'V: 


2.  Is  your  organization  part  of  the  ... 

1 . Government;  or  (SKIP  TO  Q4.) 

2.  Private  sector? 


14.7  7 


3.  Is  [STREET  NAME]  part  of  a profit  or  non-profit  organization? 

1.  Profit 

2.  Non-profit 


4.  Do  you  have  trade  unions  at  this  location? 

1.  Yes 

2.  No  (SKIP  TO  Q6) 


5.  What  percentage  of  the  non-management  employees  at  your  location  at  [STREET  NAME]  belong  to 
trade  unions? % 


6.  How  many  different  locations  exist  within  your  entire  organization  both  nationally  and  internationally? 


NOTE  TO  CATI  PROGRAMMER:  IF 
RESPONDENT  ANSWERS  DK  THEN 


OVERLAY:  ^ \ ■■  * 

6b.  Would  you  say: 


1.  Only  one 

2.  2-10 

3.  11-25 

4.  26-50 


5.51-100 
6.  101-200 

7.  201-300 

8.  More  than  300 


7.  In  how  many  countries,  other  than  the  US,  does  your  entire  organization  do  business?  That  is,  sell 
products  or  services  to  foreign  businesses  or  governments  or  participate  in  projects  with  foreign 
businesses  or  governments? 


INOTE  TO  CATIPROGRAMI 
RESPOfTOENT  ANSWERS  D1 

OVERLAY*  It* 

t - §§81 

7b.  Would  you  say: 

1.  None 

5.51-100 

2.  1-10 

6.  101-200 

3.  11-25 

7.  201-300 

4.  26-50 

8.  More  than  300 

8.  How  many  full-time  employees  work  for  your  entire  organization  at  all  locations? 


^ NOTE  TO  CATIPROGR^ 

M m * an 

wEBmmimmn 

8b.  Would  you  say: 

1.  1-25 

6.  501-1000 

2.  26-50 

7.  1001-5000 

3.51-70 

8.  5001-10,000 

4.71-100 

9.  10,001-20,000 

5.  101-500 

10.  More  than  20,000 
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9.  How  many  full-time  employees  work  at  [STREET  NAME]? 


«NOTE  TO  CATLPROGRAMMER:  IF.j 
RESPONDED  ANSWERS  DK  THEN 


;:x~\tsz#ixrxx. 

ii"  ' 


9b.  Would  you  say: 


1.  1-25 

2.  26-50 
3.51-70 
4.71-100 
5.  101-500 


6.  501-1000 

7.  1001-5000 

8.  5001-10,000 

9.  10,001-20,000 

10.  More  than  20,000 


10.  How  many  part-time  employees  work  at  [STREET  NAME]? 


4NQTE  TOCATI  PROGRAMMER:  IFfpi|^& 
RESPONDENT  ANSWERS  DK  THEN;';  I883&& 


10b.  Would  you  say: 


1.  1-25 

2.  26-50 
3.51-70 
4.71-100 
5.  101-500 


6.  501-1000 

7.  1001-5000 

8.  5001-10,000 

9.  10,001-20,000 

10.  More  than  20,000 


* * 


4 


r\ 

y 


0 


1 1.  How  many  new  employees  joined  your  location  in  the  last  12  months?  Please  include  full-time  and  part- 
time, 


janswersdkI 


lib.  Would  you  say: 


1.  1-25 

2.  26-50 
3.51-70 
4.71-100 
5.  101-500 


6.  501-1000 

7.  1001-5000 

8.  5001-10,000 

9.  10,001-20,000 

10.  More  than  20,000 


12.How  many  full-time  and  part-time  employees  worked  at  [STREET  NAME]  one  year  ago ? 


NOTE  TO  CATI  PROGRAMMER:  -'M 

RESPONDENT  ANSWERS  DK  THEN  llfpllfi 
OVERLAY:  , 


12b.  Would  you  say: 

1.  1-25 

2.  26-50 
3.51-70 
4.71-100 

5.  101-500 


6.  501-1000 

7.  1001-5000 

8.  5001-10,000 

9.  10,001-20,000 

10.  More  than  20,000 


13.Have  you  downsized  or  right-sized  your  workforce  in  the  last  five  years? 

1.  Yes,  a great  deal 

2.  Yes,  somewhat 

3.  No 


14. To  what  extent  does  [STREET  NAME]  use  independent  contractors?  Would  you  say, 

1.  Not  at  all 

2.  To  a limited  extent 

3.  To  some  extent 

4.  To  a moderate  extent 

5.  To  a great  extent 


15.How  often  does  [STREET  NAME]  use  independent  contractors... 

1 . Never 

2.  Rarely 

3.  Sometimes 

4.  Often 

5.  Always 


16. What  was  the  annual  revenue  at  [STREET  NAME]  last  year? 


NOTE  TO  CATI  PROGRAMMER:  IF 
RESPONDENT  ANSWERS  DK  THEN  OVERLAY: 


16b.  Would  you  say: 


1.  Less  than  $10,000 

2.  $10,000-$100,000 

3.  $100,000-$1  million 

4.  $ 1 million-$  10  million 


5.  $10  million-$25  million 

6.  $25  million-$  100  million 

7.  $100  million  -$500  million 

8.  $500  million  - $ 1 billion 

9.  More  than  $1  billion 


17  . What  was  the  annual  revenue  for  your  entire  organization  last  year? 


17b. 

Would  you  say: 

1. 

Less  than  $10,000 

5. 

$10  million-$100  million 

2. 

$10,000-$  100,000 

6. 

$100  million  -$1  billion 

3. 

$ 100,000-$  1 million 

7. 

More  than  $1  billion 

4. 

$1  million-$  10  million 

18.How  many  levels  of  formal  management  positions,  including  your  chief  administrator,  are  in  your  entire 
organization?  If  you  have  an  organizational  chart,  please  use  it  to  answer  this  question. 

NUMBER  OF  LEVELS  (SKIP  TO  Q19) 

(IF  DON’T  KNOW  GO  TO  1 8a). 

1 8a.  Can  you  estimate  the  range  of  management  levels  across  different  departments,  divisions,  or 

other  units  in  your  organization? 

RANGE  OF  LEVELS 


19.Has  [STREET  NAME]  removed  layers  of  management  across  several  different  departments  or  functions 
during  the  last  five  years? 

1.  Yes 

2.  No 


20.  Which  of  the  following  written  documents  exist  at  [STREET  NAME]? 


Contracts  of  employment 
An  organizational  chart 
Job  descriptions 

Manuals  of  procedures  and  regulations 
Policy  manuals 

Records  of  employees’  performance 
Records  of  employees’  time  or  hours  worked 


(ENTER  1 FOR  YES,  AND  2 FOR  NO) 

0 

0 

0 

0 

0 

0 

0 


NOTE  TO  CATI  PROGRAMMER:  ONLY  ASK  Q21  FOR  THE  ITEMS 
RESPONDENT  ANSWERED  YES  TO  IN  Q20 

DO  NOT  ASK  Q2I  FOR  THE  LAST  2 ITEMS  IN  Q20  NO  MATTER  HOW 
THEYANSWER 

21.  To  what  extent  do/does  (DOCUMENT)  dictate  how  work  is  performed 

at  [STREET  NAME]?  Would  you  say... 

1.  Not  at  all 

2.  To  a limited  extent 

3.  To  some  extent 

4.  To  a moderate  extent 

5.  To  a great  extent 

Contracts  of  employment 

o 

An  organizational  chart 

() 

Job  descriptions 

o 

Manuals  of  procedures  and  regulations 

0 

Policy  manuals 

() 

22. How  many  different  job  titles,  with  different  functions,  exist  at  [STREET  NAME]? 


. NUMBER  OF  TITLES  (SKIP  TO  Q23) 
(IF  DON’T  KNOW  GO  TO  22a.) 


22a.  How  many  different  job  titles,  with  different  functions,  exist  in  your  entire  organization ? 


NUMBER  OF  TITLES 


23.How  frequently  do  employees  at  [STREET  NAME]  experience  substantial  changes  in  their  job  duties? 

1.  Never 

2.  Rarely 

3.  Sometimes 

4.  Often 

5.  Very  often 

6.  All  the  time 


24.During  the  past  five  years,  approximately  how  many  distinct  new  types  of  jobs  have  been  created  at 
[STREET  NAME]. 


NOTE’ 

i: 

OVERL 

•> 

VDENT  ANawtiu.  1/KTHEN  gg  gg  gg 

24b. 

Would  you  say: 

1. 

None 

2. 

1-5 

3. 

6-20 

4. 

21-50 

5. 

51-100 

6. 

More  than  100 
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25 .During  the  past  five  years,  how  many  times  has  your  location  gone  through  a major  reorganization ? 

1.  Never  (SKIP  TO  Q28) 

2.  1 

3.  2 

4.  3 

5.  4 

6.  5 

7.  6 or  more 


26.  How  long  has  it  been  since  your  location’s  most  recent  reorganization?  Would  you  say... 

1 . Less  than  6 months 

2.  Between  6 months  and  1 year 

3.  Between  1 and  2 years 

4.  Between  2 and  5 years 


27 .During  the  past  five  years,  how  many  times  has  your  location’s  organizational  chart  been  revised?  Was  it... 

1.  Never 

2.  1 

3.2 

4.3 

5.4 

6.  5 or  more 


28.Is/Are  (ACTIVITY)  performed  by  your  employees  at  [STREET  NAME]? 


(ENTER  1 FOR  YES , 2 FOR  NO) 

Public  relations  and  advertising 
Sales 

Customer  service 

Transporting  or  carrying  outputs  and  resources  from  place  to  place 

Employment  or  human  resources  activities 

Training  and  developing  human  resources 

Purchasing 

Controlling  inventory 

Maintenance  or  construction 

Accounting 

Production  control 

Inspection  or  quality  control 

Design  and  development  of  products  and  processes 

Administration 

Market  research 


0 

() 

() 

o 

o 

o 

o 

o 

o 

o 

o 

o 

0 

() 

■() 


29.0n  a 1 to  5 scale,  where  1 is  not  at  all  standardized  and  5 is  completely  standardized,  rate  the  extent  to 
which  the  following  activities  are  carried  out  according  to  specified  procedures?  On  the  1 to  5 scale,  how 
standardized  is/are... 


Quality  control  inspections 
Financial  control  or  budgets 
Operational  control 
Purchasing 
Planning 

Sharing  information  with  employees  or  managers 

Recruiting  new  employees 

Selecting  new  employees 

Job  evaluation 

Performance  appraisal 

Salary  review 

Promoting  employees 

Training 

Research  and  development 


(CODE  1 THROUGH  5 - CODE  6 FOR  NA) 

() 

() 

() 

() 

() 

() 

() 

() 

() 

() 

() 

() 

0 

() 


30.To  what  extent  do  nonsupervisory  employees  at  [STREET  NAME]  have  the  authority  to... 

A.  Monitor  data  on  quality,  costs,  waste,  and  productivity?  Would  you  say... 

1.  Not  at  all 

2.  To  a limited  extent 

3.  To  some  extent 

4.  To  a moderate  extent 

5.  To  a great  extent 

B.  Determine  work  flow?  Would  you  say... 

1.  Not  at  all 

2.  To  a limited  extent 

3.  To  some  extent 

4.  To  a moderate  extent 

5.  To  a great  extent 

C.  Invest  in  new  equipment  and  technology?  Would  you  say... 

1 . Not  at  all 

2.  To  a limited  extent 

3.  To  some  extent 

4.  To  a moderate  extent 

5.  To  a great  extent 

D.  Develop  new  products,  services,  and  procedures?  Would  you  say... 

1.  Not  at  all 

2.  To  a limited  extent 

3.  To  some  extent 

4.  To  a moderate  extent 

5.  To  a great  extent 

E.  Select  new  work  group  members?  Would  you  say... 

1 . Not  at  all 

2.  To  a limited  extent 

3.  To  some  extent 

4.  To  a moderate  extent 

5.  To  a great  extent 


31. What  percentage  of  employees  at  [STREET  NAME]  are  routinely  provided  with  the  following  types  of 
information? 

A.  Information  about  your  overall  financial  results. % 

B.  Information  about  their  unit’s  financial  results. % 

C.  Advance  information  about  new  technologies  that  may  affect  them. % 

D.  Information  on  your  business  plans  and  goals. % 

E.  Information  on  competitors’  performance. % 


NOTE  TO  CATI  PROGRAMMERr  Q31 A-E.  IF  RESPONDENT 
ANSWEg&D^ 


Would  you  say: 

1. 

0% 

2. 

1-20% 

3. 

21-40% 

4. 

41-60% 

5. 

61-80% 

6. 

81-99% 

7. 

100% 

32.To  what  extent  does  [STREET  NAME]  use  formal  teams  to  accomplish  its  goals?  Would  you  say... 

1 . Not  at  all  (WE  DON’T  HAVE  TEAMS)  (SKIP  TO  Q35E.) 

2.  To  a limited  extent 

3.  To  some  extent 

4.  To  a moderate  extent 

5.  To  a great  extent 


33.To  what  extent  are  employees  in  your  location  accountable  for  the  performance  of  their  team?  Would  you 
say... 

1.  Not  at  all 

2.  To  a limited  extent 

3.  To  some  extent 

4.  To  a moderate  extent 

5.  To  a great  extent 


34  .Approximately  what  percentage  of  your  employees  at  [STREET  NAME]  work  in  intact  teams ? By  the  term 
intact  teams,  we  mean  groups  of  3 or  more  employees  who  are  jointly  responsible  for  whole  work 
processes  and  work  toward  shared  goals. % 


35.  At  your  location,  do  you  have  the  following  types  of  work  teams?  Are  there... 

A.  Functional  teams  that  are  intact  and  members  have  similar  skills  and  expertise? 

1.  Yes  (OVERLAY) 

2.  No 

B.  Cross-functional  teams  that  are  intact  and  members  have  different  skills  and  expertise? 

1.  Yes  (OVERLAY) 

2.  No 

C.  Management  teams,  with  three  or  more  senior  managers  who  make  policies  and  operational 
decisions  for  all  parts  of  your  location  or  the  entire  organization? 

1.  Yes  (OVERLAY) 

2.  No 

D.  Project  or  development  teams  that  are  temporary  and  are  brought  together  to  conduct  research 
and  development,  and  to  develop  new  products  or  services? 

1.  Yes  (OVERLAY) 

2.  No 

E.  Quality  or  improvement  teams  that  are  temporary  and  recommend  ways  to  improve  work 
procedures  or  solve  specific  problems? 

1.  Yes  (OVERLAY) 

2.  No 


NdmT&<&& PROGRAMMER:  IT  RK  YESi|i 

TOlQ3iS^^  "IHEN  OVERLAY  WITH  0§5lM 
35b.  What  percentage  of  employees  work  in  (TEAM)? 


IFttESPONPENTlAN^ 
35b.  Would  you  say: 


1. 

0% 

2. 

1-20% 

3. 

21-40% 

4. 

41-60% 

5. 

61-80% 

6. 

81-99% 

7. 

100% 

36.Do  you  have  a formal  mission  statement? 

1.  Yes 

2.  No 


37. How  many  department  or  division  heads  at  [STREET  NAME]  are  required  to  set  quantitative  annual 
goals? 

1.  None 

2.  Few 

3.  Some 

4.  Most 

5.  All 


3 8. Last  year,  did  your  management  set  and  publicize  to  all  employees  at  least  one  quantitative  performance 
goal? 

1.  Yes 

2.  No 


39. What  percentage  of  the  managers  at  [STREET  NAME]  are  either  expected  to  set  or  given  performance 
goals  for  their  unit  or  area  of  responsibility  each  year? % (IF  NONE  SKIP  TO  Q41) 


NOTE  TO  C ATI  PROGRAMMER:  IF  RESPONDENT  ANSWERS  DK 
THEN  AN O VERIAY/C  UP  READI^  : ■?' 

39b.  Would  you  say: 

1.  0%  (SKIP  TO  Q41) 

2.  1-20% 


3.  21-40% 

4.  41-60% 

5.  61-80% 

6.  81-99% 

7.  100% 


40.  What  percentage  of  the  managers  at  your  location  are  allowed  to  negotiate  their  own  goals  with  their 
supervisors? % 


NOTE  TO 
THEN  AN 

CATI PROGRAMM 
OVERLAY  GOMES 

lER-IFRES 

UPREADE 

PONDENT  ANSWERS  DK  't'  > 

40b.  Would  you  say: 

i. 

0% 

2. 

1-20% 

3. 

21-40% 

4. 

41-60% 

5. 

61-80% 

6. 

81-99% 

7. 

100% 

41  .What  percentage  of  non-management  employees  at  this  location  are  either  expected  to  set  for  themselves 
given  one  or  more  individual  performance  goals  each  year? % 


NOTElrC!  Gati^ 


THEN  AN  OVERLAY  COMES  UP  1 & 


41b.  Would  you  say: 

1.  0%  (SKIP  TO  Q43) 

2.  1-20% 

3.  21-40% 

4.  41-60% 

5.  61-80% 

6.  81-99% 

7.  100% 


42.  What  percentage  of  non-management  employees  at  your  location  are  allowed  to  negotiate  their  own  goals 
with  their  supervisors? % 


NOTETOt 
THEN  AN  < 

:ati  PROGRAM 
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42b.  Would  you  say: 

i. 

0% 

2. 

1-20% 

3. 

21-40% 

4. 

41-60% 

5. 

61-80% 

6. 

81-99% 

7. 

100% 

43. How  many  formal  performance  reviews  are  managers  required  to  give  each  employee  in  a two  year  period ? 
Only  count  standard  reviews,  not  probationary  or  initial  reviews  for  new  employees 


44 .Does  [STREET  NAME]  or  your  entire  organization  offer  formal  training  programs  to  employees? 

1.  Yes 

2.  No  (SKIP  TO  Q53.) 


45.  Which  of  the  following  training  methods  are  used  in-house  at  [STREET  NAME]?  Do  you  use... 

(ENTER  1 FOR  YES,  2 FOR  NO) 


Case  study  exercises 

Conference  method,  or  group  discussions 
Lectures  with  questions 
Business  games 

Machine  simulators,  like  flight  simulators 

Films  or  videos 

Interactive  videos 

Workbooks 

Role  plays 

Computer-assisted  instruction 
Audiocassettes 


() 

() 

() 

() 

() 

() 

() 

() 

() 

0 

() 


4 6. How  many  of  the  training  programs  used  at  [STREET  NAME]  incorporate  a careful,  systematic  training 
needs  analysis?  Would  you  say... 

1.  None 

2.  Few 

3.  Some 

4.  Most 

5.  All 


£ 


47 .How  often  are  the  training  programs  used  at  [STREET  NAME]  systematically  evaluated  to  determine 
whether  or  not  they  are  effective?  Would  you  say... 

1.  Never 

2.  Rarely 

3.  Sometimes 

4.  Often 

5.  Very  often 


48.  In  which  of  the  following  content  areas  does  your  location  provide  formal  training?  Do  not  include  on- 
the-job  or  mentoring  activities. 


Diversity 

Team  Skills 

Quality  Control  Skills 

Basic  Business  and  Economics 

Problem  Solving  Skills 

Leadership  Skills 

Customer  Service 


(ENTER  1 FOR  YES,  2 FOR  NO) 

() 

() 

() 

() 

() 

() 

() 


49a.  Approximately  what  percentage  of  employees  at  this  location  attend  no  company  sponsored  training 
courses  each  year?  % 

49b.  One  company  sponsored  training  course  each  year?  % 

49c.  Two  or  more  company  sponsored  training  courses  each  year?  % 


Q49a,b,c.  NOTE  TO^GATT^^  MUST  SUM  •?£ 

TO  100% ^EFORfelNTER^ 


50.  Does  [STREET  NAME]  have  a formal  program  for  improving  product  or  service  quality,  such  as,  TQM 
or  similar  programs? 

1.  Yes 

2.  No  (SKIP  TO  Q51) 

50a.  What  percentage  of  employees  have  received  training  in  quality  control  or  quality 
improvement  methods? % 


5 1 . Approximately  what  is  your  location’s  annual  training  budget? 
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hundred 

thousand 

million 


52.  Approximately  what  percentage  of  your  employee  payroll  is  spent  on  training  and  development? 

% 

53. Do es  [STREET  NAME]  have  programs  designed  to  support  continuous  learning  on  the  part  of  employees, 

like,  cross-training  or  job  rotation? 


54.Does  [STREET  NAME]  provide  financial  assistance  to  individuals  who  wish  to  pursue  job-relevant 
training  outside  the  organization  at  a local  college,  university,  or  vocational-technical  school? 

1.  Yes 

2.  No 


55.  Which  of  the  following  statements  best  describes  the  job  rotation  policies  for  employees  at  your  location? 
Would  you  say,,, 

1 . There  is  no  job  rotation 

2.  Rotation  within  work  groups 

3.  Rotation  within  and  across  work  groups  in  the  same  department 

4.  Rotation  across  work  groups  and  departments 


56.Does  your  location  have  a formal  recruitment  or  staffing  plan  in  place? 

1.  Yes 

2.  No  (SKIP  TO  Q59) 


57.  Which  of  the  following  types  of  data  do  you  routinely  collect  to  aid  in  evaluating  and  improving  your 
recruitment  process?  Do  you  collect  data  on... 


1.  Yes 

2.  No 


(ENTER  1 FOR  YES,  2 FOR  NO) 


Total  budget  for  recruiting  operations 
Total  recruiting  budget  per  individual  hired 

Number  of  recruiting  leads  generated  by  each  recruiting  source  used 
Number  of  prospects  hired  for  each  recruiting  source  used 


0 

0 

0 

0 


Success  after  hire  of  individuals  identified  through  various  recruiting  sources 
Average  amount  of  time  that  elapses  between  stages  in  the  recruitment  process. 


() 

() 


58.At  [STREET  NAME],  are  recruiters  trained  or  instructed  to  provide  realistic  information  to  job 
candidates  concerning  available  jobs? 


59.  Does  [STREET  NAME]  have  one  or  more  formal  orientation  programs  for  groups  of  new  employees? 

1.  Yes 

2.  No 

60. Does  [STREET  NAME]  have  a formal  mentoring  program? 

1.  Yes 

2.  No 

61.  Does  [STREET  NAME]  have  any  formal  selection  systems,  such  as,  tests  or  interviews? 

1.  Yes 

2.  No  (SKIP  TO  Q66) 

62. How  many  of  the  selection  systems  now  in  place  at  your  location  are  based  on  formal  job  analyses?  Would 

you  say... 


1 . None 

2.  Few 

3.  Some 

4.  Most 

5.  All 


63. How  many  of  the  selection  systems  used  in  your  location  are  made  up  of  procedures  that  have  been 
carefully  researched  and  shown  to  be  related  to  job  success?  Would  you  say... 

1 . None 

2.  Few 

3.  Some 

4.  Most 

5.  All 


1.  Yes 

2.  No 


64. To  what  extent  does  [STREET  NAME]  spend  money  on  research  designed  to  produce  data  to  help  you 
make  business  decisions?  Would  you  say... 

1.  Not  at  all 

2.  To  a limited  extent 

3.  To  some  extent 

4.  To  a moderate  extent 

5.  To  a great  extent 


65.  To  what  extent  does  [STREET  NAME]  use  research  and  data  to  make  organizational  decisions?  Would 
you  say... 

1.  Not  at  all 

2.  To  a limited  extent 

3.  To  some  extent 

4.  To  a moderate  extent 

5.  To  a great  extent 


ERIC 


66.  What  percentage  of  the  jobs  at  [STREET  NAME]  have  their  pay  rates  determined  or  adjusted  based  on 
formal  job  evaluation  studies? % 


NOTE5TO  CAT1  PROGRAMMER:  IF  RESPONDENT  ANSWERS  DKilf® 
THENAN:OVE^AYI&6mES'IM^ 

66b.  Would  you  say: 

i. 

0% 

2. 

1-20% 

3. 

21-40% 

4. 

41-60% 

5. 

61-80% 

6. 

81-99% 

7. 

100% 

67.  What  percentage  of  the  jobs  at  [STREET  NAME]  have  their  pay  rates  determined  or  adjusted  based  on 
comparisons  with  similar  jobs  in  other  organizations? % 


NOTE  TO 
THEN  AN 

C ATT  PROGRAMMER:  IF  RESPON 
OVERLAY  COMES  UP  READING: 

• • - • • • , 

DENT  ANSWERS  DK 

r .Vs'  l").  ,vV''.V- 

67b.  Would  you  say: 

i. 

0% 

2. 

1-20% 

3. 

21-40% 

4. 

41-60% 

5. 

61-80% 

6. 

81-99% 

7. 

100% 

O 

ERIC 
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68. Approximately  what  percentage  of  the  employees  at  [STREET  NAME]  have  a compensation  package  that 
includes  the  following  elements? 


Salaried  pay 

Profit  or  Gain  Sharing 

Knowledge  or  skill-based  pay 

Pay  based  on  individual  performance 

Pay  based  on  team  performance 

Pay  based  on  customer  satisfaction 

Pay  based  on  job  tenure/seniority 

Pay  based  on  job  attributes  such  as,  hazards  involved 


% 

% 

or  Hay  points  assigned  % 


69.  Approximately  what  percentage  of  the  employees  at  [STREET  NAME]  have  a compensation  package  that 
includes  the  following  benefits? 

Stock  ownership  in  the  organization  % 

Retirement  plan,  such  as,  401  (k),  or  a pension  plan  % 

Major  medical  insurance  % 

Life  insurance  ■ % 

Disability  insurance  % 

Flexible  working  hours  % 

Daycare  % 

Paid  leave,  including,  holidays,  vacation  time,  and  maternity  leave  % 


tgSfe.: ' G ATI  PROGRAMMER  - ' 

\ v . NO  RANDOM  STARTy  ^ J 


70.  Next,  I am  going  to  read  a list  of  organizational  values.  These  values  are  expressed  in  terms  of  how 

organizational  employees  should  behave  or  what  attitudes  are  appropriate  for  employees  to  hold.  For  each 
of  these  values,  please  evaluate  the  extent  to  which  the  value  is  characteristic  of  your  location.  Use  a 1 to  7 
point  scale,  where  1 least  characterizes  your  location  and  7 is  characterizes  your  location. 


Security  of  employment 

Risk-taking 

Flexibility 

Analytical  orientation 

People  orientation 

Fairness 

Competitiveness 

Collaboration 

Adaptability 

Predictability 

Innovation 

Social  responsibility 

Quality 

Results  orientation 


Tolerance 

Taking  advantage  of  opportunities 

Customer  service  orientation 

Action  orientation 

Stability 

Autonomy 

Attention  to  detail 

Team  orientation 

Sharing  information  freely 

Willingness  to  experiment 

Aggressiveness 

Precision 

Achievement  orientation 
Supportiveness 


Thank  you  for  answering  the  organizational  questions. 


? 


Cl. 


As  [NEGOTIATOR]  discussed  with  you,  we  would  like  you  to  help  us  gather  information  about  your 
employees’  job  responsibilities.  To  do  this,  we  need  you  to  serve  as  a liaison  or  Point  of  Contact  between  us 
and  your  organization’s  employees.  I would  now  like  to  give  you  the  list  of  occupations  that  we  want  to 
study  and  the  number  of  employees  that  we  need  for  each  occupation.  First,  I’d  like  to  verify  that  you  have 
at  least: 


; CATI  PROGRAMMER:  REPEAT  THE  FOLLOWING  FOR  AS  MANY  OCCUPATIONS  THAT  ARE - 


CIA.  (NUMBER1)  (OCCUPATION1)  at  [STREET  NAME],  Is  that  correct? 

1.  Yes  (SKIP  TO  C1B.) 

2.  No 

C 1 Aa.  How  many  (OCCUPATION  1 ) do  you  have? 

CIB.  We  will  send  you  (NUMBER  SAMPLED  OR  MAX  NUMBER  OF  EMPLOYEES)  questionnaires  {disks} 
for  (OCCUPATION!). 


In  a couple  of  days  we  will  Fed-ex  you  a package.  This  package  will  contain: 

• A copy  of  the  occupation  list  we  just  reviewed  so  you  know  which  occupations  we  are  studying. 

• An  information  sheet  about  how  to  select  participants  and  coordinate  the  data  collection. 

• A management  information  sheet  and  copies  of  the  fact  sheet  that  you  can  distribute  to  management  or  the 
employees’  supervisors  at  your  discretion.  The  letter  introduces  the  study  and  explains  the  importance  of 
employee  participation. 

• .You  will  also  get  (TOTAL  NUMBER  OF  EMPLOYEES  ABOVE)  envelopes  containing  study  introductions 

and  job  questionnaires  [floppy  disks  with  job  questions]  f one  for  each  employee  we  listed. 

• Finally,  you  will  get  a postage  paid  envelope  addressed  to  Westat,  so  that  you  can  return  questionnaires  when 
employees  have  finished. 


For  each  occupation,  we  would  like  you  to  select  the  individuals  who  will  participate.  Then  we  would  like 
you  to  pass  out  the  questionnaires  { disks] , collect  them  from  employees  as  they  finish  completing  them,  and  mail 
them  back  to  Westat  in  the  postage  paid  envelope  that  we  are  sending. 

A project  representative  from  Westat  will  call  you  in  about  a week  to  see  how  everything  is  going  and  to 
answer  and  any  questions  that  you  might  have. 

In  the  meantime,  if  you  have  any  questions,  please  contact  the  Westat  Project  Coordinator,  Katy  Erickson  at 
1-800-937-8281  x4454.  Alternatively,  if  you  have  access  to  electronic  mail  through  the  Internet,  you  may  contact 
Ms.  Erickson  at  ONET@Westat.com.  Thanks  again  for  participating  in  this  important  study. 


1503 


Appendix  B 

Data  Collection  Materials 


Exhibit  1.  Screening  interview 


ORGANIZATION  ID# 

DOT 

SCREENER  FOR  BASELINE  STUDY 


Hello,  my  name  is , and  I’m  calling  on  behalf  of  the  U.S.  Department  of  Labor  and  the  Utah 

Department  of  Employment  Security  from  Westat,  Inc.  We  are  a research  firm  in  Rockville,  MD  and  we  are  preparing  an 
important  nationwide  study  to  revise  the  Dictionary  of  Occupational  Titles. 

1.  Have  I reached  (NAME  OF  BUSINESS  ON  RIS)? 


YES  (PRIMARY  NAME  MATCH) 1 (Q4) 

YES  (SECONDARY  NAME  MATCH) 2 (Q4) 

BUSINESS  CHANGED  NAME 3 

NO,  ANOTHER  BUSINESS 4 

RESIDENCE  ONLY  (NOT  A BUSINESS) 5 (Q9) 


2.  What  is  the  name  of  your  business? 


RESIDENCE  ONLY  (NOT  A BUSINESS) (Q9) 

3.  Is  this  business  the  same  as  (NAME  OF  BUSINESS  ON  RIS)?  [PROBE:  Do  you  consider  it  the  same  business?] 

YES 1 

NO 2 (Q5) 

4.  Are  you  located  at  (ADDRESS  ON  RIS)?  [IF  P.O.  BOX,  OBTAIN  ADDRESS  AND  NOTE  ON  RIS.] 

YES 1 (Q15) 

NO 2 (Q13) 

5.  Are  you  located  at  (ADDRESS  ON  RIS)?  [IF  P.O.  BOX,  OBTAIN  ADDRESS  AND  NOTE  ON  RIS.] 

YES 1 

NO 2 (Q7) 

6.  Do  you  know  what  happened  to  (NAME  OF  BUSINESS  ON  RIS)? 

CLOSED/OUT  OF  BUSINESS 1 (CODE  SI) 

MOVED 2 (Q8) 

DON’T  KNOW 3 (CODE  NL) 

7.  Do  you  know  anything  about  (NAME  OF  BUSINESS  ON  RIS)  at  (ADDRESS  ON  RIS)? 

CLOSED/OUT  OF  BUSINESS 1 (CODE  SI) 

DON’T  KNOW 2 (CODE  NL) 


Exhibit  1.  Screening  interview  (continued) 

8.  Do  you  know  the  current  address  and  phone  number  for  (NAME  OF  BUSINESS  ON  RIS)? 


YES 1 (Q12) 

NO 2 (CODE  NL) 

9.  Are  you  located  at  (ADDRESS  ON  RIS)? 

YES 1 

NO 2 (CODE  NL) 

10.  Do  you  know  what  happened  to  (NAME  OF  BUSINESS  ON  RIS)? 

CLOSED/OUT  OF  BUSINESS 1 (CODE  SI) 

MOVED 2 

DON’T  KNOW 3 (CODE  NL) 

11.  Do  you  know  the  phone  number  or  address  of  (NAME  OF  BUSINESS  ON  RIS)? 

YES 1 

NO 2 (CODE  NL) 

12.  What  is  the  phone  number  or  address  of  (NAME  OF  BUSINESS  ON  RIS)? 

PHONE  NUMBER:  ( ) 


ADDRESS:. 


13.  Does  (NAME  OF  BUSINESS  ON  RIS)  have  an  office  at  (ADDRESS  OF  BUSINESS  ON  RIS)? 


YES 1 

NO,  MOVED  FROM  (ADDRESS  ON  RIS) 2 (Q15) 

NO 3 (CODE  NL) 


14.  Can  you  give  me  the  telephone  number  for  that  location? 

YES 1 


( ) ; 

NO 2 (CODE  NL) 


ADDRESS: 


15.  Are  there  5 or  more  people  employed  at  [ADDRESS  ON  RIS]? 


YES 1 

NO ; 2 (CODE  I) 
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Exhibit  1.  Screening  interview  (continued) 

16.  We  would  like  to  send  some  information  regarding  this  study  to  your  human  resources  director  or  the  person  who 
is  most  knowledgeable  about  organizational  and  job  information  at  (LOCATION  ON  RIS). 

What  is  that  person’s  ... 


name?: 


title?: 

Mr., 

Ms., 

Dr. 

address?: 

direct  phone  number?:  ( 

) 

fax  number?:  ( 

> 

Different  location. 


CLOSING:  To  verify  that  I have  spoken  to  someone  at  this  business,  could  I please  have  your  name? 


Thank  you  for  your  assistance. 


ERIC 


Exhibit  2.  Negotiation  Script 


[ASK  TO  SPEAK  WITH  NAME  ON  RIS.  IF  PERSON  NO  LONGER  THERE  ASK  FOR  PERSON  IN 
CHARGE  OF  HIS/HER  JOB] 

Hello,  my  name  is and  I am  calling  on  behalf  of  the  U.S.  Department  of  Labor 

{WESTAT  STAFF:  from  Westat,  an  independent  research  firm,  located  in  Rockville,  MD}  regarding  a study  to 
revise  and  computerize  the  Dictionary  of  Occupational  Titles.  The  completed  version  will  be  called  0*NET.  The 
0*NET  project  team  sent  you  a letter  about  this  important  study  about  a week  ago. 

Ql.  Did  you  get  that  letter? 

YES 1(SKIP  TO  Q2JQMT) 

NO : 2 

Qla.  I would  be  happy  to  fax  you  a copy  of  the  letter  after  our  conversation.  What  is  your  fax 
number? 


( ) (SKIP  TO  Q2JNT) 

NO  FAX (NF) 

Q 1 b.  I can  send  it.'  What  is  your  address? 


ADDRESS: 


By  computerizing  and  expanding  the  DOT  we  hope  to  make  it  more  useful  to  employers  like  yourself: 
We  will  add  more  detailed  information  about  potential  employees. 


Exhibit  2.  Negotiation  Script  (continued) 


In  order  to  do  this,  we  need  to  collect  information  from  you  about  your  organization.  We  would  like  to 
have  a representative  from  Westat  call  you  back  in  a couple  of  weeks.  {OAFC  STAFF:  Westat  is  a private 
research  firm  in  Rockville,  MD  who  will  be  helping  us  to  collect  information. } The  representative  will  interview 
you  for  about  45  minutes  on  information  about  your  organization.  Most  of  the  topics  are  listed  in  the  letter  from 
the  0*NET  project  team.  The  Westat  representative  is  a trained  professional  interviewer.  Any  information  that 
you  provide  the  interviewer  will  remain  completely  confidential  and  neither  you  nor  your  company  will  be 
individually  identified  in  any  way. 

After  this  45  minute  interview  with  you,  we  will  need  to  collect  information  from  up  to  30  of  your 
employees  about  their  jobs.  Westat  will  send  paper  and  pencil  questionnaires  {on  computer  disk}  that  employees 
can  fill  out  at  their  convenience.  These  questionnaires  will  take  about  an  hour  of  their  time.  It  would  be  helpful  if 
you  would  allow  time  in  their  workday  to  complete  the  surveys.  Again  the  information  they  provide  is  completely 
confidential,  we  don’t  even  need  to  know  their  names. 

We  would  like  you  to  act  as  a Point  of  Contact  to  organize  this  effort.  We  need  you  to  pass  out 
questionnaires  to  selected  employees,  collect  them,  and  send  them  back  to  Westat  in  a postage  paid  envelope. 
Depending  on  the  number  of  employees  needed,  you  may  also  be  asked  to  randomly  select  participants. 

Finally,  we  will  send  you  a few  brief  organizational  values  surveys  to  pass  out  to  and  collect  from  mid-  to 
upper-management  in  your  organization.  These  surveys  only  take  about  5 minutes  to  complete. 

Q2.  Would  you  be  willing  to  participate  in  this  study? 


YES  1 

NO 2(SKIP  TO  CLOSE2) 


Q2a.  Would  you  be  willing  to  coordinate  the  distribution  and  collection  of  job  questionnaires 
to  your  employees? 


YES 1 

NO 2(SKIP  TO  CLOSE2) 


Q2b.  Would  you  be  willing  to  allow  employees  time  in  their  workday  to  participate  in  this 
study? 


ERIC 


Exhibit  2.  Negotiation  Script(continued) 


Q3_INT.  Great!  Thank  you  very  much.  I just  need  to  verify  that  you  employ  people  in  several  occupations.  I am 
going  to  read  you  a short  list  of  occupations.  Please  tell  me  whether  or  not  you  have  full-time  people 
currently  working  in  these  occupations  at  vour  location.  As  I read  the  list,  please  answer  yes  if  you  do 
and  no  if  you  don’t.  Then,  I will  ask  you  how  many  people  are  currently  employed,  full-time  in  each 
occupation,  at  your  location. 


* PRINTED  FROM  DATABASE^  I 

| E 

do  you  employ 

ly  V time  at  this  locational  *&;  '*/, 

(OCCUPATION  1)? 

i 

2(b) 

(OCCUPATION  2)? 

l 

2(c) 

(OCCUPATION  3)? 

i 

2(d) 

(OCCUPATION  4)? 

l 

2(e) 

(OCCUPATION  5)? 

i 

2(f) 

(OCCUPATION  6)? 

l 

2(g) 

(OCCUPATION  7)? 

l 

2(h) 

(OCCUPATION  8)? 

l 

2(0 

(OCCUPATION  9)? 

l 

2 (j) 

(OCCUPATION  10)? 

l 

2 (k) 

(OCCUPATION  11)? 

l 

2(1) 

(OCCUPATION  12)? 

l 

2 (m) 

(OCCUPATION  13)? 

l 

2 (n) 

(OCCUPATION  14)? 

l 

2(o) 

(OCCUPATION  15)? 

l 

2 (Q5_INT.) 

1510 


Exhibit  2.  Negotiation  Script  (continued) 

Q5_INT.  Thank  you  for  verifying  the  occupations.  We  will  enter  this  information  into  a database  and  use  it  to 
select  which  occupations  we  want  information  on.  A Westat  interviewer  will  call  you  in  a couple  of 
weeks.  The  interviewer  will  do  the  45  minute  organizational  interview  with  you  over  the  phone.  He 
or  she  will  also  give  you  the  list  of  occupations  we  want  ask  your  employees  about.  This  list  will  tell 
you  how  many  employees  in  each  occupation  we  want  to  collect  information  on.  Depending  on  the 
number  we  need,  you  may  be  asked  to  pick  only  a few  of  the  employees  in  a particular  occupation  or 
all  of  them. 

Q5.  Is  there  a particular  day  of  the  week  or  time  that  is  best  for  Westat  to  call  you  back  for  the 

organizational  interview? 

ANY  DAY/TIME i. 1 (SKIP  TO  CLOSE) 

PARTICULAR  DAY/TIME 1 2 

Q5a.  BEST  DAYS  AND  TIMES  (COMPLETE  ALL  THAT  APPLY) 


CIRCLE  DAY 


CIRCLE  AM  OR  PM 


MONDAY 


AT: 


AM  / PM 


TUESDAY 


AT: 


AM  / PM 


WEDNESDAY  AT: 


AM  / PM 


THURSDAY  AT: 


AM  / PM 


FRIDAY  AT: 


AM  / PM 


SATURDAY  AT: 


AM  / PM 


SUNDAY  AT: 


AM  / PM 


CLOSE  1:  Thank  you  in  advance  for  your  help  when  the  Westat  interviewer  calls  and  again  for  the  help  you 

have  already  given  the  U.S.  Department  of  Labor.  The  interviewer  will  be  talking  with  you  soon. 


CLOSE2: 


O 

ERIC 


(USE  FOR  REFUSAL  AT  ANY  TIME)  Thank  you  for  your  time. 

- :L$12‘ 


Exhibit  2.  Negotiation  Script  (continued) 


ITEMS  INCLUDED  IN  THE  INTERVIEW 

The  purpose  of  this  study  is  to  obtain  comprehensive  and  up-to-date  information  about  organizations  and  occupations.  To  gather 
accurate  information,  a Westat  interviewer  will  be  asking  some  questions  about  your  entire  organization  and  other  questions  about  the 
location  below . 


Address: 


Some  of  the  questions  you  will  be  asked  will  be  very  specific  in  nature.  Below,  is  a list  of  questions  that  you  may  need  to  consult  your 
records  (payroll,  etc.)  to  answer.  If  you  have  not  had  time  to  do  this  when  called  for  an  interview,  the  interviewer  will  be  glad  to  call 
you  back  at  a later  time.  If  you  have  any  questions  about  the  study  or  the  information  we  are  requesting,  please  call  Katy  Erickson  at 
Westat,  1-800-937-8281  ext.  4454. 


nAt  Location 

In  Entire 

B £ Organization  > ^ 

Percentage  of  non-management  employees  belonging  to  trade  unions. 

X 

Number  of  establishments. 

X 

Number  of  countries  you  do  business  with  outside  of  the  US  . 

X 

Number  of  full-time  employees. 

X 

X 

Number  of  part-time  employees. 

X 

Number  of  full-  and  part-time  employees  one  year  ago. 

X 

Number  of  new  full-  and  part-time  employees  in  the  last  12  months. 

X 

Annual  revenue  last  year. 

X 

X 

Number  of  different  job  titles. 

X 

X 

Number  of  new  job  titles  created  in  the  last  five  years. 

X 

Amount  of  annual  training  budget. 

X 

Percentage  of  payroll  spent  on  employee  training  and  development. 

X 

Exhibit  3.  CATI  Assignment  of  Occupations 

Cl.  As  we  discussed  in  the  beginning  of  the  interview,  we  would  like  you  to  help  us  gather  information  about  your 

employees  job  responsibilities.  To  do  this  we  would  like  you  to  serve  as  a liaison  between  us  and  your  employees.  As 
(NEGOTIATOR’S  NAME)  discussed  with  you,  I would  now  like  to  give  you  the  list  of  occupations  that  we  would 
like  to  study  and  the  number  of  employees  that  we  need  for  each  occupation.  First,  I’d  like  to  verify  that  you  have  at 
least: 


r..:  - CATI  PROGRAMMER:  REPEAT  THE  FOLLOWING  FOR  AS  MANY  OCCUPATIONS  THAT  ARE  NEEDED;."- 1 


CIA.  (NUMBER1)  (OCCUPATION  1 ) at  [STREET  NAME].  Is  that  correct? 

1.  Yes  (SKIP  TOC  IB.) 

2.  No 


ClAa.  How  many  (OCCUPATION!)  do  you  have? 


C1B.  We  will  send  you  (NUMBER  SAMPLED  OR  MAX  NUMBER  OF  EMPLOYEES)  questionnaires  {disks}  for 
(OCCUPATION!). 


In  a couple  of  days  we  will  Fed-ex  you  a package.  This  package  will  contain: 

• Copies  of  a short  organizational  value  surveys  to  pass  out  to  mid-  to  upper-management  at  [STREET  NAME]. 

• A copy  of  the  occupation  list  we  just  reviewed  so  you  know  which  occupations  we  are  studying. 

• You  will  also  get  (TOTAL  NUMBER  OF  EMPLOYEES  ABOVE)  envelopes  containing  job  questionnaires  [floppy 
disks  with  job  questions },  one  for  each  employee  we  listed. 

• You  will  get  an  information  sheet  that  explains  about  selecting  participants  and  coordinating  the  data  collection. 

• Finally,  you  will  get  a postage  paid  envelope  addressed  to  Westat,  so  that  you  can  return  questionnaires  when 
employees  have  finished. 

• 

For  each  occupation,  we  would  like  you  to  select  the  individuals  who  will  participate.  Then  we  would  like  you  to  pass 
out  the  questionnaires  { disks],  collect  them  from  employees  as  they  finish  completing  them,  and  mail  them  back  to  Westat  in 
the  postage  paid  envelope  that  we  are  sending. 

Someone  from  Westat  will  call  you  in  about  a week  to  see  how  everything  is  going  and  to  answer  and  any  questions 
that  you  might  have. 

In  the  meantime,  if  you  have  any  questions,  please  call  Katy  Erickson  at  1-800- WESTAT  1.  This  is  a toll-free  call. 
Thanks  again  for  participating  in  this  important  study. 


Exhibit  4.  POC  packet 


October,  1995 
[ORG  ID] 

Dear  [POC  NAME]: 

On  behalf  of  the  0*NET  Project  Team,  I would  like  to  thank  you  for  agreeing  to  participate  in  this 
important  nationwide  project  to  gather  data  about  occupations.  The  information  that  you  and  other  employers 
provide  will  form  the  foundation  of  0*NET,  the  Occupational  Information  Network. 

0*NET  will  be  the  nation’s  primary  source  of  occupational  information.  It  will  be  a comprehensive 
database  that  provides  information  about  occupations,  worker  skills,  and  training  requirements.  0*NET  will  be  a 
useful  tool  for  employers  looking  for  qualified  people,  people  looking  for  jobs,  and  educators  preparing  people  for 
the  job  market. 

Your  role  is  critical  to  the  success  of  this  project.  We  are  asking  you  to  distribute  to  and  collect  the 
enclosed  questionnaires  from  selected  employees.  We  would  like  you  to  return  them  to  Westat,  the  company 
gathering  the  occupational  data  for  this  project,  within  two  weeks  of  receiving  this  package. 

All  information  that  we  receive  from  your  organization  is  confidential.  No  individual  or  company  will  be 
identified  in  any  statistical  summary  that  is  released  or  published. 

This  package  includes  everything  needed  to  carry  out  your  role,  including: 

• Instructions  for  Implementing  this  Project  - The  following  pages  lead  you,  step  by  step,  through 
your  role  in  the  data  collection  effort. 

• Employee  Selection  Table  - This  table  presents  a list  of  occupations  to  be  studied  and  the  number 
of  employees  needed  in  each  occupation. 

• Employee  "Job  Analysis  Questionnaires"  - Enclosed  you  will  find  questionnaires  for  each  selected 
employee.  Again,  we  ask  you  to  distribute  these  questionnaires  as  soon  as  possible  and  collect  and 
return  them  to  Westat  within  two  weeks  of  receiving  this  package.  It  should  take  each  participant 
approximately  60  to  90  minutes  to  complete  his  or  her  questionnaires. 

• Sample  Management  Information  Sheet  - You  may  find  it  helpful  to  spread  the  word  among 
department  managers  and  employees  about  this  project.  The  enclosed  Management  Information 
Sheet  provides  a sample  way  to  do  so. 

• Sample  cover  memo  to  send  to  employees  participating  in  the  project  - You  may  choose  to  use  this 
sample  cover  memo  to  send  to  employees  with  their  questionnaires.  The  memo  serves  to  introduce 
the  project  to  them,  identify  you  as  the  "Point  of  Contact"  for  [COMPANY  NAME],  and  highlight 
key  points  about  the  project  and  the  employee’s  participation. 
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Q*NET  Fact  Sheet  - This  fact  sheet  describes  the  0*NET  project.  Please  feel  free  to  copy  and 
distribute  it,  or  call  us  to  request  additional  copies. 


Large  postage  paid  return  envelopes  - Please  use  these  envelopes  to  mail  completed  employee 
questionnaires  back  to  Westat. 


A Westat  representative  will  be  calling  in  about  a week  to  see  how  things  are  going.  If  you  or  your 
employees  have  any  questions  in  the  meantime,  please  call  the  Westat  Project  Coordinator,  Katy  Erickson,  at  1- 
800-937-8281,  x4454.  Alternately,  if  you  have  access  to  electronic  mail  through  the  Internet,  you  may  contact  Ms. 
Erickson  at  ONET@Westat.com. 

Thank  you  again  for  your  participation  in  this  important  project.  Employers  and  employees  across  the 
United  States  will  benefit  as  a result  of  your  support. 


Sincerely, 


Norman  Peterson 

Project  Director,  0*NET  Project  Team 
American  Institutes  for  Research 


Enclosures 


Exhibit  4.  POC  packet  (continued) 


Instructions  for  Implementing  this  Project 


VEE  SELECTION  TABLE: 

i Employee  Selection  Table  contains  a list  of  occupational  titles  and  unique  identification  numbers.  It  also  contains  a column  titled 
“Random  Start.’*  This  will  help  with  the  selection  process  discussed  below.  The  table  is  intended  to  help  you  keep  track  of 
whom  you  selected  to  participate,  when  you  distributed  questionnaires  and  when  each  employee  returned  them.  This  table  is  for 
your  use  only  and  should  not  be  sent  back  to  Westat.  If  your  table  lists  an  occupation  more  than  once,  please  select  as  many 
employees  in  that  occupation  as  listed.  As  you  select  employees  for  participation,  please  write  his  or  her  name  in  the  space 
adjacent  to  his  or  her  occupation. 

TNG  EMPLOYEES: 

i se  refer  to  the  list  of  occupations  in  the  Employee  Selection  Table  to  be  studied  and  the  number  of  employees  in  each  occupation. 
For  some  occupations,  the  number  of  employees  needed  may  be  less  than  the  total  number  of  full-time  employees  currently 
working  in  that  occupation  at  your  location.  If  this  is  the  case,  a sample  of  employees  in  those  occupations  must  be  drawn  to 
ensure  a complete  range  of  responses.  Please  use  the  procedure  outlined  below  to  sample  employees. 

Sampling  - 


1 . Generate  and  number  an  alphabetical  list  of  all  employees  who  work  in  the  first  occupation  listed  on  the  Employee 
Selection  Table  at  your  location. 

2.  Divide  the  total  number  of  employees  in  that  occupation  by  the  number  of  times  that  occupation  appears  on  the 
table.  Round  the  result  down  to  the  nearest  whole  number.  This  will  be  your  “sampling  interval”.  If  the  sampling 
interval  is  equal  to  1,  please  distribute  one  packet  to  each  of  the  employees  in  that  occupation  and  skip  to  step  5.  If 
not,  go  to  step  3. 

3.  The  first  employee  you  select  should  be  the  first  person  whose  last  name  starts  with  the  letter  that  appears  under 
the  column  Random  Start.  If  no  employee’s  last  name  starts  with  that  letter,  begin  with  the  employee  whose  last 
name  starts  with  the  next  closest  letter. 

4.  Count  down  the  list  the  number  indicated  by  your  sampling  interval  to  select  the  next  employee.  Continue 
selecting  employees  down  the  list,  returning  to  the  top  if  necessary,  until  you  have  the  needed  number  in  that 
occupation. 

5.  Write  each  selected  employee’s  name  under  the  column  Employee  Name  next  to  their  occupation  on  the 
Employee  Selection  Table. 

6.  Repeat  steps  1 - 5 for  each  occupation  on  the  list. 

An  example  of  how  this  technique  works  appears  on  the  following  page. 


1516; 


Example  for:  Marine  World,  Inc. 


Employee  Selection  Table 


Random 

sun 

Employee  Name 

mm 

liSSlW 

itfclil 

#1  Marine  Biologist 

J 

0999111 

#2  Marine  Biologist 

J 

0999112 

#3  Marine  Biologist 

J 

0999113 

• The  POC  generated  and  numbered  an  alphabetical  list  of  full-time  Marine  Biologists  at  her  location.  The  POC 
has  17  Marine  Biologists  at  her  location. 

Marine  Biologists  at  Marine  World.  Tnc. 


1.  Mary  Arnold 

7.  Sarah  Graham 

13.  Wilma  Persley 

2.  Jim  Brown 

8.  Dwayne  Howerton 

14.  Michael  Reeder 

3.  Philip  Cohen3 

9.  Jack  Innes 

15.  Rhonda  Spear2 

4.  Nancy  Delgato 

10.  Greg  Long* 1 

16.  Mark  Trent 

5.  Lisa  Eaton 

6.  Paul  Fisher 

11.  Anita  Myer 

12.  Paula  Norville 

17.  John  Van  Mead 

She  divided  her  17  Marine  Biologists  by  3 (the  number  of  times  the  occupation  was  listed  on  the 
Employee  Selection  Table)  and  got  5.6.  She  rounded  down  to  5 (the  lower  whole  number).  This  was 
her  “sampling  interval”. 

“Random  Start”  told  her  that  she  should  start  selecting  employees  with  the  person  whose  last  name 
began  with  J . Since  there  were  no  “J’s”  on  the  list  and  Greg  Long  was  the  next  name,  she  started 
numbering  with  him. 

She  went  down  the  list,  marking  every  5th  employee  until  she  had  3 employees  numbered. 

She  wrote  in  selected  employees’  names  next  to  the  ID  Number  on  the  table. 

1)  She  wrote  Greg  Long’s  name  next  to  ID  Number  09991 1 1 . 

2)  She  wrote  Rhonda  Spear’ s name  next  to  ID  Number  0999112. 

3)  She  wrote  Philip  Cohen’s  name  next  to  ID  Number  0999 113. 

She  checked  to  see  if  there  was  another  occupation  listed  for  which  she  needed  to  select  employees. 
In  this  case  there  wasn’t  one. 
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Selecting  Replacements  * 

If  one  of  the  employees  you  select: 

• has  been  in  his  or  her  job  for  less  than  six  months;  or 

• is  not  currently  working  at  your  location  because  of  vacation,  leave,  etc.;  or 

• can  not  participate  for  some  other  reason,  please  select  a replacement 

Then: 

Select  the  person  whose  name  appears  on  the  list  immediately  following  the  person  who  can  not  participate.  In  the 
previous  example,  Nancy  Delgato  would  fill  in  for  Philip  Cohen  if  he  could  not  participate. 

JRAGING  PARTICIPATION: 

We  realize  that  supervisors  and  other  management  personnel  within  your  organization  may  need  to  know  why  their  employees  are 
participating  in  this  study.  For  this  reason  we  have  enclosed  the  Management  Information  Sheet  that  describes  your 
organization’s  involvement  with  this  project  and  an  0*NET  Fact  Sheet  that  gives  a complete  overview  of  the  0*NET  project. 
Also,  we  have  included  a sample  Cover  Memo  to  Employees  that  you  can  send  to  employees  with  their  questionnaire  packets 
to  encourage  their  participation. 

DISTRIBUTING  QUESTIONNAIRE  PACKETS: 

Each  questionnaire  packet  is  different  and  is  intended  to  be  tailored  to  individual  occupations.  ID  Numbers  from  the 
Employee  Selection  Table  should  correspond  to  the  ED  Number  printed  on  the  label  of  each  envelope.  This  number  will  help 
you  keep  track  of  each  packet  as  employees  return  them. 


• Please  match  the  ID  Number  on  the  Employee  Selection  Table  to  the  ID  Number  printed  on  the  label  on  each 
questionnaire  packet. 

• Distribute  the  packet  with  the  corresponding  ID  Number  to  each  participating  employee  within  one  week. 

• Instruct  all  participants  to  complete  their  questionnaire  packets  and  return  them  to  you  within  two  weeks. 

• Record  the  date  you  distributed  each  packet  on  the  Employee  Selection  Table. 
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RETURNING  COMPLETED  PACKETS  TO  WESTAT: 


• Employees  should  return  packets  to  you  in  sealed  envelopes.  They  should  not  send  their  own  questionnaires  in 
to  Westat. 

• Make  sure  each  employee  signed  your  company  name  across  the  seal  of  the  envelope  to  ensure  confidentiality  of 
their  answers. 

• Record  the  date  each  packet  was  returned  on  the  Employee  Selection  Table. 

• Compile  returned  packets  for  one  week  and  use  one  of  the  business  reply  envelopes  addressed  to  Westat  to  mail 
these  to  Westat. 

• At  the  end  of  the  second  week,  use  another  business  reply  envelope  addressed  to  Westat  to  mail  the  packets  that 
have  been  returned  to  you  during  the  second  week. 

• If  you  still  have  outstanding  packets  after  two  weeks,  please  remind  the  employees  to  complete  the  packets 
within  a week. 

• At  the  end  of  the  third  week,  use  a final  business  reply  envelope  to  return  the  last  of  the  employee  packets  to 
Westat. 

• Keep  the  Employee  Selection  Table  for  your  records.  Please  do  not  send  this  back  to  Westat 

A Westat  representative  will  call  you  in  about  a week  to  see  if  you  need  any  help  with  data  collection.  If  you  have 
any  questions  before  then,  please  call  the  Westat  Project  Coordinator,  Katy  Erickson,  at  1-800-937-8281  ext  4454. 
Alternatively,  if  you  have  access  to  electronic  mail  through  the  Internet,  you  may  contact  Ms.  Erickson  at 
ONET@Westat.com.  Thanks  again  for  your  assistance  with  our  data  collection  efforts.  Many  people  across  the  U.S.  will 
benefit  from  the  development  of  this  new  Occupational  Information  Network. 


ERIC 
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TABLE  1 


OCCUPATIONS  SELECTED 
AND  NUMBER  OF 
EMPLOYEES  NEEDED 
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Potential  Communication  Tools  for  the  Project's  Point  of  Contact 

To  help  make  your  role  as  Point  of  Contact  easier,  we  have  developed  some  tools  that  you  can 
use  for  communicating  to  your  organization  about  the  0*NET  project. 

Management  Information  Sheet.  You  may  find  it  helpful  to  spread  the  word  among  department 
managers  and  employees  about  this  project.  The  attached  sample  Management  Information  Sheet, 
entitled  "Important  Information  about  a Nationwide  Department  of  Labor  Project"  is  one  way  to  do  so. 
Here  are  some  possible  ways  to  use  it: 

• Simply  fill  in  your  name  and  extension  in  the  blanks  on  the  bottom  of  the  page.  Copy  it 
onto  your  organization's  letterhead. 

(If  you  prefer,  retype  this  Management  Information  Sheet,  substituting  your 
organization's  name  wherever  the  words  "our  organization"  appears.  Copy  it  onto  your 
organization's  letterhead.) 

• Post  the  Management  Information  Sheet  on  bulletin  boards  throughout  your  facility. 

• Post  it  or  have  copies  available  in  employee  kitchens,  the  cafeteria,  or  the  mailroom. 
Consider  inserting  it  into  employees'  mailboxes. 

• Consider  reformatting  it  and  turning  it  into  a memo  to  department  heads.  Have  it  come 
from  yourself,  as  the  project's  Point  of  Contact,  or  perhaps  another  executive  in  your 
organization. 

• If  a newsletter  publication  date  is  approaching  soon,  consider  submitting  it  for  inclusion 
in  your  organization's  newsletter. 

Cover  Memo  to  Employees.  We  also  have  another  tool  you  can  use  if  interested:  a sample  cover 
memo  to  send  to  employees  with  their  questionnaires.  This  memo  serves  to  introduce  the  project  to  them, 
identify  you  as  the  Point  of  Contact  for  your  organization,  and  highlight  key  points  about  the  project  and 
the  employee’s  participation.  Attached  please  find  a sample  memo  you  might  like  to  use  for  this  puipose. 

Simply  fill  in  information  such  as  your  name  and  date  in  all  the  blanks  provided,  and  then  copy 
and  distribute  the  memo.  If  you  prefer,  retype  it,  substituting  your  organization's  name  wherever  the 
words  "our  organization"  appears. 

Q*NET  Fact  Sheet.  This  fact  sheet  describes  the  0*NET  project.  Please  feel  free  to  copy  and 
distribute  it,  or  call  us  to  request  additional  copies. 


If  you  have  questions  about  any  of  these  materials  or  their  suggested  uses,  please  call  the  Westat 
Project  Coordinator,  Katy  Erickson,  at  1-800-937-8281,  x4454. 
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Important  Information  about  a Nationwide  Department  of  Labor  Project 


Our  organization  has  been  selected  to  participate  in  a nationwide  project,  sponsored  by  the  U.S. 
Department  of  Labor  and  supported  by  several  companies  from  private  industry  and  state  government 
organizations,  to  develop  an  easy-to-use  occupational  database  called  0*NET,  the  Occupational 
Information  Network. 

0*NET  will  become  the  nation's  primary  source  of  occupational  information.  It  will  be  a 
comprehensive  database  that  provides  important  information  about  occupations,  worker  skills,  and 
training  requirements. 

0*NET  will  help  employers  like  us  fmd  the  right  people  for  jobs  within  our  organization,  and  it 
will  help  our  employees  meet  their  career  development  needs. 

The  0*NET  project  team  is  gathering  occupational  data  that  will  provide  the  foundation  of 
0*NET.  They  have  asked  us  to  participate  in  this  project,  and  we  have  agreed  to  do  so.  We  are  providing 
information  about  some  of  the  occupations  within  our  organization. 

It  is  necessary  for  some  of  our  employees  to  complete  questionnaires  about  their  jobs.  These 
employees  have  been  randomly  selected  to  participate,  so  no  one  person  or  group  will  be  singled  out.  If 
someone  in  your  workgroup  has  been  chosen  to  participate,  please  allow  time  in  his  or  her  workday  to 
complete  their  questionnaires. 

Here  are  some  important  facts  about  our  involvement  in  this  project: 

• Employees'  responses  are  completely  confidential.  The  data  collected  will  be  used  for 
research  purposes  only.  No  information  about  any  individual  or  organization  will  be 
released  to  anyone  outside  the  0*NET  project  team. 

• The  questionnaires  do  not  ask  employees  for  any  information  about  their  supervisors  or 
other  individuals  in  our  organization. 

• No  sensitive  or  confidential  information  is  asked. 

• The  questionnaires  take  from  60  to  90  minutes  to  complete. 

• A report  containing  summary  information  about  this  project  will  be  published  early  next 
year. 


If  you  would  like  to  learn  more  about  this  project  or  our  organization's  participation  in  it,  please 
call at . 


Thank  you  for  your  support  on  this  important  project! 
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MEMORANDUM 

TO:  Project  Participant 

FROM:  

DATE:  

SUBJECT:  Participation  in  U.S.  Department  of  Labor  Occupational  Research  Project 


Our  organization  has  been  selected  to  participate  in  a nationwide  project,  sponsored  by  the  U.S. 
Department  of  Labor  and  supported  by  several  companies  from  private  industry  and  state  government 
organizations,  to  develop  an  easy-to-use  occupational  database  called  0*NET,  the  Occupational 
Information  Network. 

0*NET  will  become  the  nation's  primary  source  of  occupational  information.  It  will  be  a 
comprehensive  database  that  provides  important  information  about  occupations,  worker  skills,  and 
training  requirements.  0*NET  will  help  employers  like  us  find  the  right  people  for  jobs  within  our 
organization,  and  it  will  help  our  employees  like  yourself  meet  their  career  development  needs. 

The  0*NET  project  team  is  gathering  occupational  data  that  will  provide  the  foundation  of 
0*NET.  They  have  asked  us  to  participate  in  this  project,  and  we  have  agreed  to  do  so.  We  are  providing 
information  about  some  of  the  occupations  within  our  organisation 

A scientific,  random  sample  of  employees  in  organizations  across  the  country  has  been  selected 
to  participate  in  this  project.  You  have  been  selected  to  be  one  of  the  participants.  Your  role  is  to  provide 
key  information  about  the  kind  of  work  you  do  by  completing  the  attached  questionnaires. 

I am  the  designated  "Point  of  Contact"  for  our  organization  for  this  project.  My  role  is  to 
distribute  to  and  collect  questionnaires  from  our  employees  who  are  participating  in  the  project. 

Attached  please  find  your  questionnaires.  It  describes  the  0*NET  project  and  includes  a set  of 
Job  Analysis  Questionnaires  for  you  to  complete  and  return  to  me.  The  questionnaires  ask  you  to  provide 
information  about  your  occupation.  The  questionnaires  should  take  about  60  to  90  minutes  to  complete. 
Please  look  the  questionnaires  over  right  away  and  at  earliest  opportunity  complete  them  and  return  them 
to  me  by . 

Finally,  I want  to  stress  that  all  the  information  you  provide  will  be  confidential.  It  won't  be 
viewed  by  your  supervisor,  company  management,  or  any  other  employee  for  any  purpose.  The 
information  you  provide  is  for  research  purposes  only. 

If  you  would  like  to  leam  more  about  this  project  or  have  questions  about  your  participation, 
please  call  me  at 

Thank  you  for  your  support  on  this  important  project ! 
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Exhibit  5.  Incumbent  packet 


October,  1995 

Dear  Project  Participant: 

You  are  being  asked  to  participate  in  a very  important  nationwide  project.  A team  of  public  and 
private  sector  organizations,  sponsored  by  the  U.S.  Department  of  Labor,  is  developing  an  easy-to-use 
occupational  database  called  0*NET,  the  Occupational  Information  Network.  The  database  will  help 
millions  of  employees  like  yourself,  employers,  educators,  and  students  make  informed  decisions  about 
careers,  training,  education,  and  work. 

0*NET  will  assist  individuals  in  identifying  the  right  skills  and  training  needed  to  achieve  their 
career  goals.  0*NET  will  also  assist  employers  in  selecting,  training,  and  placing  qualified  people  in 
jobs.  0*NET  will  equip  you  with  the  knowledge  and  informational  tools  necessary  to  compete  and 
succeed  in  today's  competitive  economy. 

We  are  contacting  organizations  nationwide  and  asking  employees  like  yourself  to  provide  key 
information  about  the  type  of  work  they  do.  The  data  you  are  being  asked  to  provide  will  become  the 
foundation  of  0*NET.  Although  your  participation  is  completely  voluntary,  we  are  counting  on  your 
support.  Ultimately  the  successful  development  of  0*NET  depends  upon  your  responses  to  the  questions 
being  asked  regarding  jobs  and  the  skill  and  training  requirements  necessary  to  perform  those  jobs. 

Your  responses  will  remain  completely  confidential.  The  information  you  provide  will  be  used 
for  research  purposes  only.  Data  will  be  used  to  describe  occupations,  not  specific  jobs  or  the  people 
performing  them.  Your  answers  will  not  be  seen  by  anyone  in  your  organization  and  will  not  affect  your 
employment  within  your  organization  in  any  way.  No  individual  or  company  will  be  identified  in  any 
statistical  summary  that  is  released  or  published. 


Thank  you  for  your  support.  Without  your  help,  the  development  of  0*NET  would  not  be 
possible! 


Sincerely, 


Norman  Peterson 

Project  Director,  0*NET  Project  Team 
American  Institutes  for  Research 


P.S.  Attached  is  an  information  sheet  which  lists  the  contents  of  this  package  and  explains  how  easy  it  is 
to  participate  in  this  important  nationwide  project. 


Enclosures 
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Exhibit  5.  Incumbent  packet 


IMPORTANT  INFORMATION  FOR  PROJECT  PARTICIPANTS 


This  Package  Contains  the  Following  Materials: 

• Letter  from  the  Q*NET  project  team  - This  letter  was  sent  to  the  person  in  your  organization  who 
is  serving  as  the  "Point  of  Contact"  between  the  0*NET  team  and  employees  like  yourself.  This 
person  has  already  agreed  to  support  your  participation  in  this  project. 

• Q*NET  Fact  Sheet  - This  fact  sheet  describes  0*NET,  the  Occupational  Information  Network,  and 
the  project  in  which  you  are  being  asked  to  participate. 

• Job  Analysis  Questionnaires  - These  questionnaires  ask  you  to  provide  background  information 
and  to  answer  questions  about  various  aspects  of  your  occupation. 

• Envelope  with  your  Point  of  Contact’s  name  on  it  - Please  use  the  same  envelope  in  which  you 
received  your  questionnaires  to  return  completed  questionnaires  to  your  Point  of  Contact. 


Here's  How  Simple  It  Is  To  Participate  in  this  Worthwhile 

Project! 

The  enclosed  questionnaires  ask  you  to  provide  valuable  information  about  your  occupation.  The  entire 

process  should  take  you  about  60  to  90  minutes. 

All  you  need  to  do  is  follow  these  3 easy  steps... 

(1)  Read  the  instructions  for  each  questionnaire  and  answer  each  question.  Complete  the 
questionnaires  within  2 weeks. 

(2)  Place  the  completed  questionnaires  in  the  0*NET  envelope  and  seal  it.  Sign  your  company  name 
across  the  sealed  flap  of  the  envelope  to  ensure  that  your  envelope  will  remain  unopened  until  it 
reaches  Westat,  the  research  company  evaluating  the  data. 

(3)  Return  the  sealed  envelope  to  your  Point  of  Contact. 


If  you  have  any  questions  or  are  missing  any  of  the  package  materials,  please  call  the  Westat  Project 
Coordinator,  Katy  Erickson,  at  1-800-937-8281,  x4454. 


Thank  You  for  Your  Help! 


Exhibit  5.  Incumbent  packet 
September  5, 1996 
Dear  Employer, 

On  behalf  of  the  U.S.  Department  of  Labor,  we  request  your  assistance  in  an  important 
nationwide  research  project  to  develop  a powerful  occupational  database  called  0*NET,  the 
Occupational  Information  Network. 

0*NET  will  become  the  nation's  primary  source  of  occupational  information.  It  will  be  a 
comprehensive  database  that  provides  important  information  about  occupations,  worker  skills,  and 
training  requirements.  Essentially,  0*NET  will  help  employers  find  the  right  people  and  people  find  the 
right  jobs. 

0*NET  is  being  developed  by  the  0*NET  project  team,  which  consists  of  the  Department  of 
Labor,  American  Institutes  for  Research,  and  several  other  private  and  public  sector  organizations.  The 
enclosed  Fact  Sheet  gives  you  more  specific  information  about  the  0*NET  project. 

Your  organization,  along  with  many  other  small,  mid-sized,  and  large  businesses  across  the 
United  States,  has  been  selected  to  participate  in  this  important  research  project  by  a scientific  and 
random  process.  In  a recent  telephone  call  to  your  organization,  you  were  identified  as  the  person  most 
knowledgeable  about  occupational  information  within  your  organization. 

A member  of  the  0*NET  project  team  will  call  you  within  the  next  two  weeks  to  discuss  the 
details  of  this  research  project.  Subsequently,  an  interviewer  from  Westat  Inc.,  the  company  responsible 
for  gathering  the  occupational  data  for  this  project,  will  contact  you  to  ask  some  questions  about  your 
organization.  The  final  phase  of  the  project  involves  collecting  job  analysis  occupational  information 
from  a sample  of  employees  in  your  organization.  All  the  information  that  you  and  your  organization's 
employees  provide  will  be  completely  confidential  and  will  only  be  used  in  composite  form  with  the 
information  obtained  from  other  companies.  No  individual  person  or  company  will  be  analyzed  or 
reported  in  any  way.  * 

While  participation  is  completely  voluntary,  we  strongly  urge  your  assistance  in  this  important 
project.  Your  help  provides  the  building  blocks  needed  to  develop  the  0*NET  database,  so  that  people 
like  you  can  use  and  benefit  from  this  timely  database  of  occupational  information. 

If  you  have  any  questions  about  this  project,  please  contact  the  Westat  Project  Coordinator,  Katy 
Erickson,  at  1-800-937-8281,  x4454.  Alternately,  if  you  have  access  to  electronic  mail  through  the 
Internet,  you  may  contact  Ms.  Erickson  at  ONET@Westat.com. 

Thank  you  in  advance  for  your  help  on  this  important  project.  Employers  and  employees  across 
the  United  States  will  benefit  as  a result  of  your  participation. 


Sincerely, 


Norman  Peterson 

Project  Director,  0*NET  Project  Team 
American  Institutes  for  Research 


Exhibit  6.  Follow-up  interview  (continued) 


Q4.  Did  you  distribute  the  assigned  questionnaire  packets  or  {disks } to  all  of  the  employees  that  you 
selected  in  each  occupation? 


YES  (All) 1 (Q5) 

NO  (Not  All) 2 (Q4a) 


Q4a.  Please  refer  to  the  list  of  selected  occupations  that  we  sent  to  you.  How  many  employees 
out  of  the  requested  number  in  each  occupation  listed  did  not  receive  a questionnaire 
packet  or  { disk } ? 


[R  MAY  REFER  TO  PACKET  ID#,  IF  SO,  PLEASE  WRITE  IT  DOWN  - TRY  TO 
GET  OCCUPATION  ALSO] 


Exhibit  6.  Follow-up  interview 


[REFER  TO  OCCUPATION  LIST.  ASK  TO  SPEAK  WITH  POC  (FOUND  ON  RIS).  IF  POC  NO 
LONGER  THERE  ASK  FOR  PERSON  IN  CHARGE  OF  HIS/HER  JOB] 

**NOTE:  Interviewers  will  be  provided  a list  of  the  occupations  and  number  of  employees  requested 
from  each  organization.  This  can  be  used  to  determine  if  all  survey  packets  are  complete. 


that  you  are  coordinating  for  the  U.S.  Department  of  Labor.  I am  calling  to  check  on  the  status  of  the 
employee  questionnaire  packets  or  {disks}  and  to  see  if  you  have  any  questions.  You  may  need  to  refer 
to  you  copy  of  table  1 for  this  discussion. 

(NOTE:  IF  THERE  IS  ANY  QUESTION  THAT  YOU  CANNOT  ANSWER,  TELL  THE 
RESPONDENT  TO  CALL  KATY  ERICKSON  OR  ANGIE  RASMUSSEN  AT  1-800-937-8281.  TRY 
TO  GET  ANSWERS  TO  ALL  QUESTIONS  FIRST) 

Q 1 . Did  you  receive  the  package  of  survey  materials  from  Westat? 


Q2.  Were  you  able  to  randomly  select  the  requested  number  of  employees  in  each  occupation? 


Hello,  my  name  is 


.,  calling  from  Westat  about  the  employee  survey 


NO 


YES 


1(Q2) 

2 (P-MAILOUT) 


YES 

NO  2 (Q2a) 


1(Q3) 
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Exhibit  6.  Follow-up  interview  (continued) 


Q2a.  What  kinds  of  problems  did  you  have  with  random  selection?  [ALSO  READ  P-SLCTN 
AFTER  RECORDING  RESPONSES  TO  Q2a] 

1.  One  or  more  selected  employee(s)  was  unavailable,  but  I still  used  random  selection 

(Q3) 

2.  My  organization  did  not  have  as  many  employees  in  the  occupation  as  was  needed,  so  I 
selected  all  employees  in  that  occupation  (Q3) 

3.  Didn’t  feel  like  doing  random  selection  (Q2b) 

4.  Have  too  many  employees  to  do  random  selection  (Q2b) 

5.  The  process  was  too  confusing  (Q2b) 

6.  Other  (Specify):  (Q2b) 


P-SLCTN  I may  ask  one  of  my  supervisors  to  call  you  in  regard  to  your  difficulties  with  the  selection 
process  so  that  we  might  get  some  information  that  will  help  us  to  improve  the  process  for 
future  data  collection.  Let’s  go  on. 

Q2b.  If  you  were  not  able  to  use  random  selection,  did  you  distribute  the  survey  packets  or 
{disks}  to  the  requested  number  of  employees  in  each  occupation  listed  anyway? 


YES 1 (Q3) 

NO 2 (Q2c) 

I USED  RANDOM  SELECTION 3 (Q3) 

Q2c.  Did  you  select  any  employees  to  participate  in  the  study? 

YES 1 (Q3) 

NO 2 (P-NOEES) 


Q3.  How  many  employees  did  you  select  altogether? 

employees  selected  total  (Q4) 

NONE  (P-NOEES) 


Exhibit  6.  Follow-up  interview  (continued) 


Q4b,  Why  didn’t  these  employees  receive  questionnaire  packets  or  {disks}? 


OCCUPATION 
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Exhibit  6.  Follow-up  interview  (continued) 

Q5.  According  to  your  copy  of  Table  1,  how  many  employees  have  returned  questionnaire  packets 
{disks}  to  you? 

returned  packets  (Q6) 

None (CLOSE2) 

Q6.  How  many  employees  have  not  returned  their  packets  or  {disks}  to  you? 

outstanding  packets  (CLOSE2) 

None  (Have  all  packets) (Q7) 

Q7.  Have  you  mailed  the  packets  or  {disks}  back  to  Westat? 


YES 1 (CLOSE  1) 

NO  2 (CLOSE  1) 


P-MAILOUT:  I will  check  with  our  survey  distribution  staff  to  find  out  why  you  did  not  receive  a 
package.  You  should  be  receiving  a package  within  the  next  week.  We  will  call  you 
again  in  a couple  of  days  to  see  how  things  are  going.  Thanks!  (END  CALL) 

P-NOEES:  Have  supervisor  call  them  back.  (END  CALL) 

CLOSE  1:  Great!  The  U.S.  Department  of  Labor  really  appreciates  your  assistance  with  this 

important  project.  I am  looking  forward  to  receiving  the  employee  surveys  from  you  in 
the  mail  shortly.  Thanks  again.  (Q8) 

CLOSE2:  As  soon  as  you  receive  all  of  the  surveys  from  your  employees,  please  send  them  to 

Westat  in  the  business  reply  envelope  that  we  sent  you.  I am  looking  forward  to 
receiving  the  employee  surveys  from  you  in  the  mail  shortly.  I want  to  thank  you  again 
on  behalf  of  the  U.S.  Department  of  Labor  for  all  of  your  help  with  this  important 
project.  (Q8) 
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Exhibit  6.  Follow-up  interview  (continued) 


Q8.  Do  you  have  any  other  questions? 


YES 1 

NO 2 (end  call) 


(RECORD  AND  CODE  QUESTION)  [IF  CANT  ANSWER,  GO  TO  SUPVSR] 


SUPVSR  I do  not  have  the  resources  to  answer  your  specific  question.  I will  ask  someone  on  the 
project  supervision  staff  to  call  you  with  more  information.  Or,  if  you  like,  you  can  call 
Katy  Erickson  or  Angie  Rasmussen  directly  at  1-800-937-8281.  (END  CALL) 


U.S.  Department  of  Education 

Office  of  Educational  Research  and  Improvement  (OERI) 
National  Library  of  Education  (NLE) 
Educational  Resources  Information  Center  (ERIC) 
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